Eval, Testing, and Debugging

Eval, Testing, and Debugging is 2.6% of the CCDV-F blueprint — roughly 1 or 2 of the 53 scored questions, making it the smallest domain on the exam. Its structure tells you exactly how to prepare for it: the blueprint gives it a single sub-skill, **Debugging and Error Handling**, carrying the domain's entire 2.6%. Evaluation methodology and test strategy are in the domain's name but not in its decomposition, so questions here are far more likely to hand you a broken production behaviour than to ask you to design a rubric.

That makes this a domain you should study last and study narrowly. The material overlaps heavily with Applications and Integration, which already covers the error envelope, retry classification and `stop_reason` branching — so most of the preparation is free if you have done that domain properly. What is distinctive here is the diagnostic mindset: when a Claude-backed system misbehaves, the failure is usually not where an engineer's instinct points. Empty output is rarely an API fault; it is a truncation, a refusal, or a content block the caller never looked at. Intermittent tool failures are rarely model quality; they are an unanswered tool call or results split across messages. A cache that never hits is never a caching bug; it is a byte changing in the prefix.

Approach the domain as symptom-to-cause pattern matching. For each of the half-dozen ways these systems fail, know the observable symptom, the field that reveals the real cause, and the fix. That is a short list, and it is worth an hour.

What the exam actually tests

• —Reading the error envelope: HTTP status, `error.type`, and the `request_id` that makes support tractable
• —Classifying failures as retryable (429, 5xx, connection) versus terminal (400, 401, 403, 404, 413)
• —Diagnosing empty or truncated output via `stop_reason` rather than assuming an API fault
• —Debugging the tool loop: unanswered `tool_use` blocks, mismatched `tool_use_id`, results split across turns
• —Using `usage` fields — including cache read and write counts — as the primary instrumentation
• —Why non-determinism changes what reproduction means, and why sampling parameters are not the answer

Read the error envelope before changing anything

Every non-2xx response carries a structured envelope: an HTTP status, an `error.type` string, a message, and a `request_id`. The type strings map predictably — `invalid_request_error` (400), `authentication_error` (401), `permission_error` (403), `not_found_error` (404), `request_too_large` (413), `rate_limit_error` (429), `api_error` (500), `overloaded_error` (529). The first three of those are caller defects and will fail identically forever; the last three are transient. The official SDKs expose one typed exception class per status, so branch on the class rather than matching substrings in a message — message text is not a stable interface. Log the `request_id` on every failure: it is the identifier that lets a support conversation trace a specific call end to end.

Empty or short output is a stop_reason question

The most common false alarm in production is 'the model returned nothing'. Almost always the response was fine and the caller looked in the wrong place. Check `stop_reason` first. `max_tokens` means your own output cap truncated the answer — raise it, or stream. `refusal` means safety classifiers declined: the call succeeded with HTTP 200, content may be empty or partial, and `stop_details` carries the category. `tool_use` means the model is waiting on you and the loop simply never continued. `pause_turn` means a server-side tool loop paused and the request needs resuming. A related trap: on current models thinking blocks are returned with empty text by default, so a UI rendering reasoning shows a long silence unless summarised display is explicitly requested.

Debugging the tool loop

Agentic failures concentrate in a handful of loop defects. An assistant turn containing several `tool_use` blocks must be answered by a single user message containing a `tool_result` for each — miss one and the conversation is malformed; split them across messages and you quietly train the model out of parallel calls. Every `tool_result` must carry the `tool_use_id` from the block it answers. A failing tool still needs a result with `is_error: true` and an actionable message, because silence gives the model nothing to recover from. And every loop needs a bound: an iteration cap, so a tool that keeps returning something the model finds unsatisfying cannot spin indefinitely.

Instrument with usage, not with intuition

The `usage` object on every response is the cheapest observability you have. `input_tokens` and `output_tokens` show where spend actually goes — and on a cached workload, `input_tokens` reports only the uncached remainder, so total prompt size is that plus `cache_creation_input_tokens` plus `cache_read_input_tokens`. That decomposition is the diagnostic for the classic 'caching does nothing' report: if `cache_read_input_tokens` stays at zero across requests that ought to share a prefix, something upstream is changing bytes — a timestamp in the system prompt, a per-user tool list, non-deterministic serialisation. Pair usage logging with the request ID and the model ID actually used, and most cost and latency mysteries resolve without a debugger.

Testing a non-deterministic dependency

A model call is not a pure function, so classical reproduction does not apply: the same input can produce different output, and a defect that appears once in twenty runs is still a defect. Two adaptations follow. First, capture enough to replay — the exact model ID, every request parameter, the full message array, and the request ID — because 'it worked on my machine' is meaningless when the prompt was assembled dynamically. Second, replace spot checks with a small graded set. A few dozen representative inputs with objective pass criteria, run before and after every prompt or model change, will catch regressions that eyeballing three outputs never will. Where correctness is judgement-shaped, grade with explicit criteria rather than a vague quality score.

Where candidates go wrong

How to study this domain

This is the smallest domain on the exam — 2.6%, one or two questions — so cap your time here and study it last. Nearly everything it needs you will already have from Applications and Integration; treat this as a focused review rather than new material.

The highest-value hour is spent building a symptom-to-cause table and keeping it to a single page. Empty response, truncated response, intermittent tool failure, runaway loop, cache never hits, sudden cost spike, unexplained latency. For each row write the field you would inspect first — `stop_reason`, the `usage` decomposition, the `tool_use_id` pairing, the error type — and the fix. That table is essentially the whole domain.

Then do one deliberate breakage exercise: take working code, remove a `tool_result` from a parallel batch, and watch what the API says. Do the same with a truncating `max_tokens`. Ten minutes of induced failure teaches the diagnostic instinct better than any amount of reading, and it makes the distractors obvious — they are almost always the option that changes the request before anyone has read the response.

Common questions