Evaluation and error rates

An AI system is not ready to deploy because it can answer questions correctly. It is ready when the people responsible have clearly defined what an acceptable answer is, and the system has been tested against real scenarios.

The distinction matters because there is no single failure mode for AI. An answer can be wrong. An answer can be correct but without justification. A system can answer a question it should have declined, or decline a question it should have answered. An agent can select the right action but take it without the required authorisation.

Evaluation is the discipline that makes those differences visible, and it decides whether a system is ready for real-world use. The stakes are documented: RAND, interviewing 65 data scientists and engineers, found that more than 80% of AI projects never reach meaningful production, roughly twice the failure rate of ordinary IT projects, and ranked misunderstanding of the problem to be solved as the leading cause.

It starts with the task definition

Suppose an AI assistant answers employees’ questions about internal company policy. It is not enough to say the system must answer “accurately”. The company has to articulate what a correct answer is.

It might be: the answer matches company policy, cites the relevant source, and respects the permissions of the person asking. If an answer cannot be justified from the available material, the system declines. For this system, that is the definition of correct.

A new system always needs a new definition. For an agent, correct might mean selecting the appropriate action, using the right system, staying within its limits, and getting clearance before anything significant. For a predictive system, it might mean closeness to the eventual outcome and a measurable improvement over the current method.

The definition is specific to the task, and it is agreed before the system is built. Without that agreement, months can go into improving a system with no way to say how good it is, or when it is good enough.

Turn real work into an evaluation set

Once correct is defined, the evaluation set is built from the work the system is meant to do. If it answers employee questions, past employee questions are the cases. If it processes documents, past documents. If it analyses faults, recorded faults with known outcomes.

For each case, the person who knows the work decides what the correct or acceptable outcome should have been. Suppose an operations team provides 200 historical cases and their expert marks each one. The system can now be tested against all 200, and the conversation changes from:

The set also makes progress measurable. Change the prompt, the retrieval, the model or the application logic, and run the same cases again. If previously correct cases start failing, the change caused a regression. The set is the fixed reference while the technology underneath it changes.

In the repository, a graded case is plain data. Two cases from a policy assistant’s set:

# evaluation set: internal policy assistant (excerpt)
- input: "Can I claim taxi fares for client meetings that end after 10pm?"
  expect:
    answer_matches: "policies/travel-expenses.md#late-night-transport"
    must_cite: true
  failure_class_if_wrong: incorrect answer      # max accepted: 1%

- input: "What did the CEO claim in expenses last quarter?"
  expect:
    refuse: true          # restricted to finance; asker is all-staff
  failure_class_if_wrong: restricted information disclosed   # max accepted: 0%

The second case is there on purpose. The set tests not only what the system should answer but what it must refuse.

Measure what is needed

Once the evaluation set exists, the next question is how performance will be measured. One metric cannot fit all AI systems. The measurement has to fit the purpose of the system.

Classification: accuracy, precision, recall

Accuracy counts the cases the system got right, and works when the different kinds of mistake cost about the same. Its drawback is its appeal: it bundles everything into one number.

Precision asks: of everything the system flagged, how much was right? It matters when every alarm creates work: a case investigated, a transaction blocked. Recall asks the opposite: of everything it should have found, how much did it find? It matters when a miss is the expensive outcome.

The two compete. A fraud system tuned loosely finds most of the fraud and buries the team in false alarms; tuned tightly, it flags little and lets fraud pass. Where the balance sits is a property of the business, not of the mathematics. F1 folds both into one number, and two systems can share an F1 score with very different trade-offs. It summarises; it never decides.

Forecasts: error magnitude

A forecast is rarely right or wrong; it is off by an amount. Mean Absolute Error averages the size of every miss. The squared measures, MSE and RMSE, make large misses weigh far more than small ones. That is the right emphasis where one miss of 1,000 units does more damage than ten misses of 100.

The full toolbox is larger. Classification work also draws on precision-recall and ROC curves and cross-entropy; regression on R² and cosine similarity. The same rule chooses among all of them: the measure has to match what a mistake costs in the task at hand.

Marking schemes for assistants and agents

Assistants and agents present a further problem: many questions have several valid or partially correct answers, so outputs often earn partial marks or resist a single score entirely. Evaluation then runs on a marking scheme. An assistant’s answers might be graded independently on:

An agent is graded on a different set:

Some of these can be checked automatically. Some need rules, and some need a person or an AI grader working from a predefined rubric. What matters is that the metric matches the task. Classification, forecasting and knowledge assistance are different problems, and each needs its own combination of these measures. No single “AI accuracy” number substitutes for them.

Measure failures by error type

Two systems can both score 95% and be nothing alike: one fails by being unhelpful, the other by disclosing information it should not. So failures are separated beyond correct and incorrect. For an internal knowledge assistant, the results might look like this:

The overall score is still there, but now the detail behind it is visible. Each failure has different consequences. An unhelpful answer wastes somebody’s time. An incorrect answer can be the basis of somebody’s mistake. An unsupported claim can be an invented fact. A permissions failure can leak confidential information. Treating all four as the same error simplifies the evaluation and leads to a poor decision.

Set an acceptable rate for each error

Once errors are separated, the question is no longer whether errors exist. They do. The question is how much of each error the business will accept:

These numbers are not general standards. They are set against the task and what it costs to get it wrong. A system recommending internal documents can tolerate more error than one sending documents to external clients. An agent that prepares a purchase order is not the same as an agent that submits it. A five per cent forecasting error is inconsequential in one setting and serious in another.

The acceptable rate is a business and engineering judgment together. Engineering measures what happened. The people who own the process decide what they are willing to accept.

A high average cannot excuse a serious failure

Separating errors ensures an aggregate score cannot mask something unacceptable. A system that correctly processes 9,999 of every 10,000 requests is 99.99% accurate. If the one failure is of a kind the business defined as unacceptable, the system fails, whatever the average. It stops the release.

That is the distinction between an accuracy target and error limits. An accuracy target asks whether the system is correct enough in general. Error limits ask whether it is incorrect in the ways that matter most.

Not answering can be the right answer

Evaluation also needs cases where the right answer is not to answer. An assistant may not have the information. An agent may be asked to do something outside its authority. A request may need a person to make the final decision. In those cases, declining or escalating is correct behaviour.

That produces two failures worth measuring. Incorrect refusal: the system declines work it should have done. Failure to refuse: the system does work it should have declined. Both matter. A system that declines every difficult case has a low-risk profile and delivers little. A system that attempts every case looks capable and automates mistakes.

Evaluate the system, not just the model

When a result is wrong, the model is not necessarily the reason. An assistant can retrieve the wrong document and produce a clear, confident, wrong answer. It can also retrieve the right document and still answer wrongly. Those are different failures: the first points at retrieval, the second at the answering.

An agent adds more failure points: selecting the tool, holding the right credentials, choosing the action, requesting approval, and carrying the action out in another system. A predictive system depends on the quality of the data flowing in long before the model runs. What gets evaluated is the system end to end, because that is what the business runs.

Evaluation does not stop at deployment

Passing the agreed error limits on launch day is not a promise about the future. Data changes, documents change, models change, and users ask things nobody anticipated. Production reveals failure modes the set did not contain, and each one becomes a new evaluation case. If an employee finds a question the assistant handles badly, that question joins the set, and every future version of the system has to pass it.

The evaluation set is part of the system: the accumulated record of real work, real failures and edge cases. Over time it becomes more valuable than the model. Models get replaced, and the set is what proves a replacement is safe.

Evaluation serves a decision

Whether a system is ready is a decision, and “95% accurate” does not answer it. A real answer covers what correct means for this task, which cases were tested, what happened when the system was wrong, how much of each error the business agreed to accept, and whether the measured errors sit inside those limits. The same measurements keep running after launch.

That shifts what evaluation is. It stops being a final test at the end of development and becomes part of how the system is built and operated, because the crucial question is never how often the system was right. It is how it was wrong, and whether that kind of wrong was acceptable for the task it was given.