Temperature Zero Is Not Determinism: Reproducing the Incident You Can't Re-Run

A model gives a customer a wrong, expensive answer. You open the post-mortem, paste the exact prompt back into the exact same endpoint with temperature=0, and you get a different answer. Not a worse one, not a better one — a different one. The bug you are supposed to root-cause refuses to reproduce on demand, and the one knob everyone told you guarantees reproducibility just lied to your face.

This is the moment most teams discover that "set temperature to zero" is folklore, not an engineering control. Temperature zero changes how the model samples — it forces greedy decoding, always taking the highest-probability token. It says nothing about whether the probabilities themselves come out the same twice. And in a production serving stack, they frequently don't.

The instinct is to blame randomness in the sampler, so people double down: set top_k=1, set top_p=1, pin the seed. The output still drifts. The reason is that the nondeterminism was never in the sampler. It was in the floating-point arithmetic that produced the logits, and — more subtly — in the batch your request happened to land in when it hit the GPU. Until you understand that distinction, your incident reviews will keep ending in a shrug.

Greedy Decoding Removes One Die, Not All of Them

A language model emits a probability distribution over the next token. Temperature scales that distribution before sampling: high temperature flattens it (more surprising tokens become viable), low temperature sharpens it, and temperature zero collapses sampling to argmax — pick the single most likely token, every time. So far, so deterministic.

The problem is what happens when two candidate tokens are nearly tied. Suppose token A scores 0.5001 and token B scores 0.4999. Greedy decoding will deterministically pick A — if the logits come out the same. But the logits are the result of billions of floating-point multiply-accumulate operations across dozens of transformer layers, and floating-point addition is not associative. (a + b) + c does not always equal a + (b + c) once rounding is involved. Reorder the additions and the tenth decimal place wobbles.

A wobble in the tenth decimal place is invisible — until it crosses the boundary between 0.5001 and 0.4999. Then argmax flips from A to B. Now you have a different token at position 103, that token conditions every token after it, and two "identical" greedy runs have visibly diverged into different paragraphs. Greedy decoding removed the randomness in which token you pick given the scores. It did nothing about randomness in the scores.

This is why the people who carefully set top_k=1 and pin the seed are surprised. They closed the front door of the sampler while the logits were leaking out the back.

The Real Culprit Is Batch Invariance, Not Concurrency

Here is the part that trips up even experienced engineers, because the obvious explanation is wrong. The obvious explanation is: "GPUs run thousands of threads concurrently, thread scheduling is nondeterministic, so the sums come out in a different order each time." Intuitive — and mostly false.

Run a single matrix multiplication on a GPU twice with the same inputs and you get bitwise identical results, every time, despite all that concurrency. Modern GPU kernels are written to accumulate in a fixed order regardless of how threads get scheduled. Concurrency alone does not buy you nondeterminism. If it did, a lone matmul would already be unstable, and it isn't.

The 2025 analysis from Thinking Machines Lab pinned the actual cause, and it is more interesting: the forward pass is run-to-run deterministic but not batch-invariant. Batch invariance means the numbers computed for your request don't depend on what else was in the batch when the kernel ran. Production servers fail this property. To use the hardware efficiently, the runtime dynamically batches concurrent requests together, and the batch size determines which kernel strategy fires — how reductions get split, how work is tiled across cores. A different batch size means a different reduction order means different rounding means, occasionally, a flipped token.

Sit with the consequence: the other users on the server change your output. When traffic is light, your request runs in a batch of two and takes one numerical path. At peak load it runs in a batch of two hundred and takes another. Same prompt, same temperature, same seed, same model weights — different answer, and the variable that changed was someone else's traffic. That is why the bug evaporates when you replay it at 2 a.m. against a quiet endpoint. You didn't fail to reproduce the conditions; the load was one of the conditions, and you can't summon it on demand.

The kernels that break invariance are the ones doing reductions across variable-sized dimensions:

• RMSNorm uses a data-parallel reduction, but small batches force a split-reduction strategy that changes the summation order.
• Matrix multiplication picks tiling and "split-K" strategies based on batch dimensions; the small-batch path reduces differently from the large-batch path.
• Attention is the worst offender, because reductions run along the feature and sequence dimensions, and KV-cache chunking choices depend on how the sequence was split across prior requests.

What Determinism Actually Costs

It is fixable, which is the genuinely useful news. The fix is to write batch-invariant kernels: pick one reduction strategy and one tiling configuration and use it regardless of batch size, so the numerics for your request are identical whether you share the GPU with one other request or a thousand. The Thinking Machines work demonstrated this end-to-end. With stock kernels, Qwen3-235B at temperature zero produced 80 unique completions out of 1,000 identical requests, with the first divergence appearing around token 103. With batch-invariant kernels, all 1,000 completions were bitwise identical.

The cost is performance, and you should know the rough magnitude before you reach for it. Forcing a fixed matmul configuration cost roughly 20% versus tuned cuBLAS. An unoptimized deterministic vLLM ran about 2.1× slower than the nondeterministic baseline — 55 seconds against 26 — and a better deterministic attention kernel brought that down to about 1.6× (42 seconds). Choosing FP32 over lower precision where stability matters more than speed costs more still. So determinism is not free, and it is not a flag you flip in production for every workload. It is a deliberate trade you make where reproducibility is worth a real slice of throughput.

There is one place the trade is unambiguously worth it: on-policy reinforcement learning. If your sampling path and your training path compute logits with even slightly different numerics, the policy you're learning from isn't quite the policy you're updating, the KL-divergence between them drifts, and training can quietly destabilize. Bitwise-identical inference makes the two paths the same path and the divergence goes to zero. If you're doing RLHF or RLVR, batch-invariant inference moves from a nice-to-have to load-bearing.

Engineering for the Post-Mortem You Can't Re-Run

Most teams don't control their kernels — they call a hosted API where batch invariance isn't on the menu. The realistic goal there isn't bitwise reproducibility; it's a post-mortem you can actually conduct. That means capturing enough state at incident time to reason about what happened, because you will not be able to re-summon the exact run.

Start with versioning, because a hosted model is a moving target. Call version-pinned endpoints, never the floating alias. An alias like "latest" can be re-pointed under you — new weights, a tweaked system prompt, a different safety filter — and your behavior shifts with no diff in your own code. A pinned build at least holds the model still. Then snapshot every other version that shapes the output: prompt template, retrieval index, tool schema, guardrail/moderation config, and eval dataset. The forensic question "what was the system, exactly, at 14:32?" should have a stored answer, not an archaeology project.

Capture logprobs on a sample of production traffic, not just final text. Token-level log-probabilities are your distribution fingerprint. If a provider silently re-points an alias, quantizes a model, or swaps hardware, the text might look fine while the probability distribution shifts measurably — and logged logprobs are often the only evidence that anything changed under you. They turn "the model feels different this week" into a chart.

Then build a replay harness that records the things you can pin and lets you override the things you can't. The classic offenders beyond inference numerics are the same primitives that have always made distributed systems hard to replay: wall-clock time, random seeds, external tool responses, and retrieval results. An agent that reasons over "now" will behave differently at replay than at the incident unless you feed it a frozen clock. Record every tool call and its response so replay reconstructs the exact execution path instead of issuing fresh, different side effects. Pin seeds even though they won't fully save you, because removing the variables you can control narrows the search for the one you can't.

The mental shift is the whole point. Stop treating an LLM call as a pure function f(prompt) → output that you can re-evaluate at will. Treat it as a distributed event whose result depended on the model build, the server load, the floating-point reduction order, the clock, and the tool responses at that instant. Some of those you can pin; some you can only record. "Set temperature to zero" controls exactly one of them — the least important one for reproducibility — which is why it disappoints everyone who leans on it. Reproducibility isn't a parameter you pass. It's an architecture you build before the incident, so that when the answer you can't re-run lands in your queue, you have something to replay.