Temperature zero is not determinism

There is a claim that appears in a great many internal design documents, and it is wrong. It reads roughly: “we set temperature to zero, so the model is deterministic and the output is reproducible.” Teams build audit commitments on that sentence. Some build reproducibility guarantees for regulators on it.

Set temperature to zero on a shared inference endpoint, send the same prompt a thousand times, and you will not get a thousand identical answers. You will get dozens of distinct ones. The sampler is not the reason.

Where the variance actually comes from

Temperature controls one source of randomness: how the next token is drawn from the probability distribution the model produces. At zero you take the highest-scoring token every time— greedy decoding, no dice. That genuinely removes sampling randomness, which is why the belief is so durable. It removes the source everyone knows about.

The source almost nobody accounts for is numerical. GPU kernels for operations like matrix multiplication and normalisation compute their results by summing many floating-point values, and floating-point addition is not associative: adding a set of numbers in a different order gives a very slightly different answer. The order those sums are performed in depends on how work is split across the hardware, and that split depends on the shape of the batch being processed.

Inference servers batch concurrent requests together for throughput. The batch your request lands in depends on how many other requests arrived at the same moment. So the reduction order changes, the arithmetic changes in the last bits, and those differences propagate forward through every layer. Most of the time they change nothing observable. Occasionally they are enough to flip which token scores highest— and once one token differs, everything after it is a different continuation.

The experiment worth repeating

Thinking Machines Lab published the clearest demonstration of this in late 2025. Running Qwen-3-8B under vLLM, they sent one thousand identical prompts at temperature zero and collected dozens of unique completions. Then they pinned the batch size to one— and the nondeterminism disappeared entirely. Re-enable dynamic batching and it came straight back.

That is the whole argument in one experiment, and it isolates the cause precisely. The model did not change. The prompt did not change. The sampler was already deterministic. The only variable was how many other requests happened to be in flight, and that was enough to change the output.

They also showed it is fixable. Kernels can be written to be batch-invariant— to use a fixed reduction strategy regardless of batch shape, so the arithmetic is identical whether your request is alone or one of two hundred. There is a working implementation. It costs throughput, because you are giving up shape-dependent optimisations that exist for good reasons. That is a real engineering trade. The point is that it should be a decision you make, not a property you discover during an incident review.

Three sources of variance, routinely treated as one

The practical problem is that teams collapse everything into a single vague belief that “the model is random”, which makes all of it feel equally unfixable. It is worth separating three distinct things, because they have different owners and different remedies.

Sampling variance is yours. It comes from temperature, top-p and seeds, it is fully under your control, and turning it off is a configuration change. This is the one everybody manages.

Numerical variance belongs to the serving infrastructure. It comes from batch shape, kernel implementation, hardware generation and server load. On a shared commercial endpoint you do not control any of those, and the provider is under no obligation to hold them stable. You cannot configure your way out of it; you can only choose infrastructure that offers batch invariance, or design so you do not need it.

Version variance is contractual. The model behind an API name changes. Weights get updated, quantisation changes, routing changes. This is the largest source of behavioural drift in practice and the one most likely to be discovered by a customer, because it can arrive overnight with no change on your side at all.

A team that says “our system is reproducible” is usually making a claim about the first and has verified nothing about the second or third.

Why this becomes a compliance problem, not just an engineering one

In regulated deployments, reproducibility is not an engineering nicety. It is evidentiary. “Show me how the system reached that decision, and demonstrate it” is a standard request from an auditor, a regulator, or opposing counsel. The honest answer for most current systems is that the decision can be explained from logs but not reproduced by re-execution— and those are very different things to have to say in a hearing.

This is why decision logging is not optional and cannot be reconstructed after the fact. If you cannot re-run the system and get the same output, then the record you captured at the time is the only account of what happened that will ever exist. It has to be complete at the moment of the decision: the inputs, the retrieved context at the version it was retrieved, the tools called, the parameters, the model identifier, and the output. A log that lets you re-derive the reasoning is worth more than a system that can be re-run but was never recorded.

Measure it before you promise it

None of the above requires you to take anyone’s word for it, including ours. The variance in your own stack is a property you can measure this week, and the measurement is straightforward enough that there is no good reason to be guessing.

Take a representative production prompt. Send it several hundred times at temperature zero, against the endpoint you actually use in production, during a period of realistic load rather than at three in the morning when your traffic is low and the batches are small. Count the distinct outputs. That number is your reproducibility rate, and for most teams running against a shared commercial endpoint it will be considerably worse than expected.

Then repeat it during peak hours and compare. If the two distributions differ, you have demonstrated the load dependency directly, on your own infrastructure, with your own prompts— which is a far more persuasive artifact in a design review than a citation to somebody else’s benchmark. Run it again after any provider announcement, since version variance and numerical variance are indistinguishable from the outside and this is the cheapest instrument you have for detecting either.

Track the number over time. A reproducibility rate that silently degrades is one of the earliest signals available that something changed underneath you, and unlike most drift detection it costs a few hundred API calls rather than a monitoring programme.

Design for bounded behaviour, not identical behaviour

The constructive move is to stop demanding a guarantee the infrastructure cannot give and start specifying one it can. You will often be unable to promise that the same input produces byte-identical output. You can almost always promise that the output stays inside a defined envelope— that a refund never exceeds a threshold, that a clinical summary never introduces a medication absent from the source, that a classification never lands outside a permitted set.

Those are properties you can enforce deterministically outside the model, with validation that runs on every output regardless of what the model produced. They are also properties you can state in a specification and test against directly, which makes them far better safety claims than any statement about accuracy. Determinism is a property of an implementation. Boundedness is a property of a design, and it survives a provider changing their kernels without telling you.

So keep temperature at zero— it removes a real source of variance and costs nothing. Just stop describing it as determinism, and stop writing commitments that depend on it being true. The number of distinct answers you get to one repeated question is a measurable property of your stack. Measure it before a regulator asks you to.