Your evaluations are not a safety case

The dashboard is green. Ninety-four percent on the internal suite, up from eighty-eight the sprint before. The harness ran nine hundred cases overnight and flagged four, all triaged as acceptable. Someone screenshots it, drops it into the launch channel, and the agent ships on Thursday.

Six weeks later it does something to a customer that nobody tested for, and the post-mortem opens with the only question anyone cares about: how did this pass?

It passed because it was never asked the right question. Half of enterprises have now deployed an AI agent or model feature that cleared internal evaluation and still caused a customer-facing failure; one in four have done it more than once. That figure comes from a self-selected sample of 157 companies, so read it as a direction rather than a decimal. The direction is not in dispute.

The reflex, once this happens, is to write more evaluations. It rarely helps, and the reason it rarely helps is structural. An evaluation is a measurement. A deployment decision is an argument. You do not arrive at the second by accumulating more of the first.

A score is a measurement. A decision is an argument.

Every mature safety-critical industry converged on the same artifact, and none of them call it a test report. They call it a safety case: a structured argument that a system is acceptably safe in a defined context, supported by evidence. The argument is the deliverable. Test results are inputs to it, in the same way that a load calculation is an input to a structural engineer’s sign-off rather than a substitute for it.

Ask a rail or medical-device team why their system is safe to operate and you get a document with a claim, a scope, an evidence chain, and a signature. Ask most AI teams the same question and you get a number. The number may be excellent. It is still not an answer, because it does not say what the system must not do, under what conditions the result holds, or what would have to be observed for the answer to change.

The number on your dashboard is the wrong number

There is a specific, measurable reason evaluation scores mislead, and it has nothing to do with sloppiness. Most suites measure the wrong random variable.

Sierra’s τ-bench work drew the distinction cleanly by separating two metrics. The familiar one, pass@k, asks whether an agent succeeded in at least one of k attempts. It rewards capability — can this system, on a good day, do the thing. The other, pass^k, asks whether it succeeded in all k attempts. It measures reliability — does this system do the thing every time it is asked.

The results were not subtle. The strongest model evaluated scored under 50 percent on a single attempt at the retail task, and roughly 25 percent across eight identical attempts — about a 60 percent relative collapse. Same task. Same model. Same prompt. The only variable was being asked more than once. Every model tested degraded as k rose.

That benchmark is now dated and the model in question has been superseded several times over, which is exactly why the structural point matters more than the specific figure: capability and consistency are different axes. A system can improve rapidly on the first while barely moving on the second, and a benchmark that reports the first tells you very little about the second. Internal suites almost universally report something closer to a single run. Your customers do not run your agent once. They run it fifty thousand times, and the distribution of those runs — not the mean — is what they experience.

Three gaps that more tests cannot close

The distribution gap comes first. An evaluation exercises inputs someone thought to write down. Production supplies the ones nobody did. Coverage measured against an imagined distribution says nothing rigorous about the real one, and in an open world the set of inputs you failed to imagine is not merely large — it is unbounded. Adding cases moves a denominator you chose. It does not touch the one that matters.

The consistency gap comes second, and it is the pass^k problem stated in business terms. A system that is right ninety-four percent of the time is not a system that is ninety-four percent safe. It is a system whose failures you have not yet characterised. Safety depends on which four percent, whether they cluster, whether they correlate with high-value transactions, and whether they repeat under load. A mean answers none of that.

The composition gap comes third and is the most commonly missed. Evaluations test a model, or an agent in isolation, in a harness built for the purpose. Production runs that agent wired to retrieval, tools, permissions, rate limits, business rules, other agents, and a queue that is deepest exactly when things are going wrong. Failures concentrate in the seams between those parts, and the seams are precisely what the harness abstracts away. The agent you evaluated and the system you deployed are not the same object.

None of these close by adding cases, because none of them are coverage problems. That is why the industry’s instinct — write more evals — buys so much less than teams expect. Effort rises, the number rises, and the relationship between the number and reality does not improve at all.

Broken gates do not make organisations careful. They make them bold.

Here is the finding that should genuinely worry anyone running an AI programme. Only five percent of organisations say they fully trust the automated evaluations that gate their release decisions. In the same population, sixty-six percent already permit some production deployment without human review, or intend to within twelve months.

Read those two numbers together, because separately they are unremarkable and together they are alarming. An organisation that does not trust its gate does not stop shipping. It stops respecting the gate. And the pattern strengthens with scale: enterprises above 2,500 employees deploy without human review more often than smaller ones, seventy percent against sixty-four, while reporting more customer-facing failures, not fewer — fifty-four percent against forty-eight.

The mechanism is ordinary organisational behaviour. A control that produces confident-sounding numbers, occasional unexplained alarms, and no account of its own limits is indistinguishable from bureaucracy. People route around it. They stop reading it. They start treating the review as a scheduling obstacle between a finished feature and a launch date. The most-cited reason for distrusting automated evaluation was not bias, explainability, or privacy — it was poor alignment with real-world outcomes, named by twenty-nine percent. The complaint is not that the tests are unfair. It is that they do not predict anything.

What a safety case actually contains

The remedy is not a better dashboard. It is a different artifact, and it has five parts that an evaluation report structurally cannot supply.

It states a claim in operational terms. Not “the agent is accurate” but “the agent will not issue a credit above five thousand euro without a second authorisation, and will not act on a customer record it cannot attribute to an authenticated session.” A claim you cannot falsify is not a claim; it is a sentiment.

It defines an operating envelope. Every claim holds under conditions — transaction volume, data freshness, latency, tool availability, input language, model version. Outside those conditions the claim is void, and the system should be built to detect that it has left the envelope rather than continuing to act with the confidence it had inside it. This is the discipline that “controlled degradation” exists to serve, and it starts with writing the boundary down.

It presents evidence together with the limits of that evidence. This is the hardest cultural shift and the one that separates assurance from marketing. An evaluation report that states only what it covered is a sales document. One that states what it could not reach — which conditions went untested, which populations are underrepresented, which failure modes the harness cannot produce — is evidence a reviewer can actually reason about.

It quantifies residual risk and names who accepts it. Not a committee, not a function, a person. Risk that everyone owns is risk nobody owns, and the single most clarifying question in any launch review is which individual’s signature appears under the sentence beginning “I accept that this system will occasionally…”

It specifies what invalidates it. A model version bump, a tool schema change, a policy update, measured drift beyond a threshold — each should trigger re-verification of the specific claims it undermines, automatically. A safety case has an expiry date. Most evaluation reports do not even have a shelf.

One question for Monday morning

You do not need a programme to start. Before the next launch review, ask a single question: what result could this evaluation have produced that would have stopped the deployment?

Then wait. If nobody can name a specific, plausible threshold — a number set before the run, that would have held against schedule pressure — the evaluation was not a gate. It was a ritual performed on the way to a decision that had already been made. That is worth knowing before a customer discovers it for you.

The follow-up is harder still. Ask someone to complete the sentence “this system is safe to deploy because…” without citing a score. The teams that can do it have a safety case, whatever they happen to call it internally. The teams that cannot have a measurement and a hope, and the distance between those is where the last six weeks of every post-mortem we have read actually went.

None of this requires inventing anything. Aviation, rail, nuclear power and medical devices have been arguing about residual risk under uncertainty for fifty years, and they converged on the same answer: state the claim, bound the conditions, show the evidence and its holes, name the accepter, define what breaks it. The discipline exists and is well understood. It has simply not been imported.

The industry is not short of evaluation. It is short of argument. That gap is where deployments fail, and it does not close by running the suite again.