Evaluation, Testing & Optimization

Evaluation, Testing & Optimization is 16% of the CCAR-P blueprint — about 10 of the 63 questions. The objectives are defining evaluation metrics across accuracy, latency, cost, safety and security; designing evaluation datasets and test frameworks with mixed methodologies; running A/B tests and iterative improvement; diagnosing system issues such as prompt failure, hallucination and model mismatch; optimising token usage, latency and cost-performance trade-offs; and monitoring performance through logging and observability tooling.

What makes it a Professional-tier domain is that every question presupposes you cannot re-run a request and get the same answer. Traditional test intuition — one input, one expected output, assert equality — does not survive that, and the candidates who struggle here are usually the ones who quietly conclude the system therefore cannot be tested.

It can. You test properties instead of strings, distributions instead of single runs, and you accept that a result carries a confidence interval. That reframe is most of the domain. The rest is diagnostic discipline: given a symptom, which layer changed — prompt, retrieval, tools, model, or the traffic itself — and what single experiment isolates it.

This domain pairs tightly with Integration. A diagnosis question is only answerable if you know what the pipeline stages are and what each one fails like, so study the two together rather than in sequence.

What the exam actually tests

• —What an evaluation set is, and choosing the cheapest grading method that detects the failure you care about
• —Sourcing and structuring a dataset so it covers the decision boundary rather than the easy cases
• —Running a comparison that survives non-determinism: sample size, one change at a time, pre-declared metric
• —Isolating the failing layer from a described symptom
• —Distinguishing structural cost waste from a model-choice problem
• —Knowing which optimisation trades quality and which does not

An evaluation set is a versioned artifact, and grading is a design choice

An eval is three things: inputs, a definition of what a good output looks like, and a grading method. The grading method is where architectural judgement shows.

Exact or structural matching is cheap and brittle, and it is right for constrained outputs — a classification label, an extracted field, a schema-conformant object. Programmatic assertions sit in the middle: does the JSON validate, does the cited document exist, is the figure within tolerance, did the run stay under the token budget. Model-graded evaluation with a written rubric is the option for open-ended output, and it is the most expensive and least stable.

The blueprint's phrase is mixed methodologies, and it is the right instinct: use the cheapest grader that can actually detect the failure you care about, and reserve model grading for the qualities nothing simpler can measure.

The dataset is where the work is

Most weak evaluation programmes have a reasonable harness and a bad dataset. Cases invented at design time reflect what the team imagined; production traffic reflects what users do, and the difference is where the failures live.

Build from real inputs, deliberately over-sample the classes you have seen fail, and keep a held-out portion you do not iterate against. Coverage of the decision boundary matters far more than volume — fifty cases spanning the hard calls beat five hundred easy ones, because a set the system already passes cannot tell you anything.

The operational half is the loop: production failures should become cases. That is the same feedback edge Solution Design asks you to put in the architecture, seen from the testing side, and a system without it accumulates known failures nobody is regression-testing against.

Comparing two variants without fooling yourself

The same request run twice can produce different output, so a single-run comparison measures noise. Three rules follow.

Run enough samples per variant that the difference you are claiming is larger than the spread you observe within a variant. Change one thing at a time — a new prompt and a new model together give a result nobody can attribute. Declare the metric before you look, because after the fact there is always some cut of the data on which the new version wins.

Two further cautions specific to this domain. A cost comparison across models must be re-measured rather than scaled, because token counts are model-specific. And an improvement measured only on the set you tuned against is memorisation, not progress — which is what the held-out set exists to catch.

Diagnosis: which layer changed

The method is to hold layers constant and vary one. The exam's published sample is the canonical shape: a RAG system starts returning confident but incorrect answers after a document refresh, with model version and latency unchanged. Everything about that stem points at retrieval, and the strong candidate does not need to reason it out because the signature is familiar.

Worth learning as signatures. Format drift or ignored instructions after nothing changed on your side points at a model or configuration change. Stale or irrelevant facts point at retrieval. Answers that stop mid-sentence point at an output limit — check how the turn ended before touching the prompt. A cost spike with flat traffic points at cache invalidation or a retry loop, not at usage.

The general rule: a symptom that appeared without a deploy is usually a dependency, an index, or a data change.

Optimisation, and which levers cost quality

Separate the levers that are free from the ones that trade.

Structurally free: fixing a prompt prefix so caching actually reads, filtering tool results before they re-enter context, bounding retries and not retrying failures that cannot succeed, and moving latency-tolerant work onto batch processing, which trades completion time for a substantially lower rate. These recover cost without touching output quality.

Trades: downgrading a model, lowering effort, shortening retrieved context, cutting the number of candidates before reranking. Each is legitimate and each needs the eval in place first, because the whole point is that you are exchanging some quality for cost or latency and you need to see the size of the exchange.

On latency, distinguish time to first token from total time. Streaming changes what a user perceives without changing what the system spends.

Where candidates go wrong

How to study this domain

Build one small evaluation set for something you own, even twenty cases, and grade it three ways: exact match, a programmatic assertion, and a model-graded rubric. An afternoon on that teaches more than any amount of reading, because you immediately see which failures each method can and cannot detect — and that mapping is what the questions test.

Then practise diagnosis as pattern recognition rather than derivation. Write a short table of symptom to likely layer: stale facts, format drift, truncation, cost spike with flat traffic, a regression right after a model change. Under time pressure you want to recognise these, not derive them.

For the optimisation objectives, learn the shape of the economics rather than the numbers: caching makes repeated prefixes dramatically cheaper to read but costs a premium to write, so it pays back after a few reuses and loses money on a one-shot prefix; batch processing trades latency for a much lower rate. Exact prices change and are not what is tested.

Finally, when a question offers an optimisation, ask what it costs in quality and whether the scenario could notice. The answer that adds measurement before the change is very often the key.

Common questions