Measuring answer quality in production

Why is one accuracy number not enough?

Consider an assistant answering questions about internal policy. It can be wrong in ways that have nothing in common with each other. It can give an answer that contradicts the policy. It can give a correct answer with no source, which is unusable in a regulated process. It can decline a question it should have answered, which erodes adoption quietly. It can answer a question it should have declined, because the person asking was not entitled to the information.

Those four failures have different consequences, different owners and different fixes. Collapsing them into a single percentage discards exactly the information needed to decide whether the system can be launched.

An accuracy figure is also meaningless without the definition behind it. Correct against what standard, measured on which cases, marked by whom. The definition has to be written before the number means anything, and it has to be specific to the task rather than borrowed from a benchmark.

Criteria appear while grading, not before

Shankar and colleagues at Berkeley named the difficulty precisely in their 2024 paper. People need criteria in order to grade outputs, and they discover their criteria by grading outputs. They called it criteria drift, and it explains why evaluation sets written in a kickoff workshop tend to be wrong.

The practical consequence is that the first version of an eval set is a draft produced by reading real outputs, and it changes as more are read. Teams that treat the set as a specification to be signed off tend to measure the wrong properties confidently.

How is the set actually built?

From the work the system is meant to do. If it answers employee questions, the cases are past employee questions. If it processes documents, past documents. If it triages faults, recorded faults with known outcomes.

Someone who understands the work marks each case with the correct or acceptable outcome. Two hundred marked cases is enough to change the conversation from an impression to a measurement, and the measurement is specific: this many resolved correctly, these failed, and here is each failure with its reason.

The failures then become a taxonomy. Not a framework’s list of generic failure modes, but the ones this system actually produces, named in the client’s own vocabulary. Each mode gets an assertion, written as code where the property is checkable and as a judge prompt only where it is not.

Failure mode	Detected by	Consequence	Owner
Contradicts policy	Comparison against the marked outcome	Wrong advice acted on	Policy owner
Correct but unsourced	Code assertion on citation presence	Unusable in a regulated process	Engineering
Declined an answerable question	Marked case set, refusal rate monitor	Quiet loss of adoption	Product
Answered beyond entitlement	Permission-aware test cases	Data exposure	Security

Using a model as a judge

Grading every output by hand does not scale, so a model is often used to grade. This is workable and frequently misused.

A judge is a measuring instrument, and an instrument that has not been calibrated produces numbers with no known relationship to the property being measured. Calibration here means grading a sample by hand, comparing the judge’s labels against those grades, iterating the judge prompt until agreement is acceptable, and recording the agreement figure. Shankar’s EvalGen work formalises this loop; Hamel Husain’s practitioner writing describes the same discipline for teams without a research budget.

Two details from that practice are worth keeping. Binary judgments are easier to keep consistent than graded scales, both between people and over time. And the agreement figure belongs next to every score the judge produces, because a pass rate of 94% from a judge that agrees with humans 70% of the time is not a measurement anyone should act on.

Why does measurement have to continue after launch?

Because the system underneath changes without notice. At least one laboratory has published a postmortem describing serving faults that degraded output quality with no change to the version string clients were pinning. Retrieval corpora grow. Prompts get edited by people who do not run the evaluation. Usage patterns shift as adoption spreads beyond the pilot group.

The response is a fixed set of questions run on a schedule, weekly in most engagements, against the live system rather than a staging copy. Between runs, cheap statistical monitors fill the gap: refusal rate, output length distribution, token consumption per request. None of these measure quality directly, and all of them move when something has changed.

Every real failure found in use is added to the set. The measurement therefore gets harder as the system meets more of the world, which is the intended behaviour. A pass rate that never falls usually means the set stopped growing.

What we hold to

Two positions govern how we build evaluation into an engagement.

A single accuracy number is not a measurement of an AI system. The report states the error taxonomy, the rate for each mode, and the consequence attached to it, because those are the terms in which a launch decision is actually made.

An LLM judge may be used where its agreement with human grading has been measured, and that agreement is reported with every score it produces. An unvalidated judge is an opinion with a number attached, and it is worse than no measurement because it is harder to argue with.