A model scored 100% on ExploitBench. That tells you about the benchmark.

Reported figures for OpenAI's Astra include 100% on ExploitBench and a 91.5% decline rate against jailbreaking attempts, alongside disclosure that the model discovered zero-day vulnerabilities during evaluation. Taken together they are a good illustration of why evaluation scores make poor deployment evidence.

A saturated benchmark has stopped measuring

When a model scores 100%, the benchmark has retired. It can no longer distinguish this system from a better one, and it cannot tell you anything about performance beyond its own distribution.

This is a routine and well-understood phenomenon, and it is still consistently misread. A saturated score is not evidence of unbounded capability. It is evidence that the test was finite and has been exhausted. What you learn is that the model handles everything ExploitBench contains — which, given the pace of these releases, was probably assembled against a threat model already a year old.

Ceiling results tell you about the ceiling, not about the system that hit it.

91.5% is a failure rate, and it is the number that matters

Presented as a safety result, 91.5% refusal reads as strong. Inverted, it is a 8.5% success rate for an adversary who is, by construction, trying repeatedly.

Security properties do not average. An attacker with an 8.5% per-attempt success rate against an automatable interface does not experience a well-defended system; they experience a brief delay. A dozen attempts gets you past it with high probability, and nothing about the interface makes a dozen attempts expensive.

Pair that with the first number. The same release documents a model that discovers exploitable vulnerabilities autonomously and can be steered into misuse roughly one time in twelve by a determined adversary. Those two facts belong in the same sentence and they are rarely printed in one.

What a safety case would require instead

This is the distinction we keep returning to. A score answers how often the system behaved acceptably on a fixed distribution. A deployment decision needs an argument: what must this system never do, under which conditions does that hold, what evidence supports it, and what would have to be observed for the argument to fail.

No published number here answers any of those. That is not a criticism of the labs, who are publishing more evaluation detail than they did two years ago, and the threshold disclosures are genuinely useful. It is a criticism of how the numbers get consumed downstream, where a benchmark result is routinely pasted into a risk assessment as though it settled something.

The practical reading

If you are considering one of these models for a security workflow, the reported scores are the wrong input. The questions that matter are which failure modes remain at 8.5%, what the blast radius is when one lands in your environment, and what detects it when the model is wrong while appearing confident.

The vendor cannot answer those, because the answers are properties of your deployment rather than of their model.

Written by the Devence Lab research team.