When the Hardware Lies: Silent Data Corruption Meets Stochastic Software

Somewhere in your inference fleet, there is probably a chip that computes wrong answers. Not crashes — wrong answers. It passed manufacturing tests, it passes health checks, and under a specific combination of instruction sequence, data values, voltage, and temperature, it returns a number that is simply incorrect. The hyperscalers have documented this at scale: roughly one device in a thousand silently corrupts data, a rate several orders of magnitude worse than the cosmic-ray bit flips we used to worry about.

For fifty years, software had an immune system against this: determinism. Same input, same output — so you could checksum, replay, and compare against golden results, and lying hardware eventually got caught. LLM inference is the first major workload where that immune system is gone. When a model gives a slightly worse answer, was it the sampler being a sampler, or a degraded GPU flipping bits in your KV cache? Nobody can tell by inspection. A flaky accelerator can quietly drag down your quality metrics for weeks while every dashboard stays green.

One in a Thousand Chips Is a Liar

The evidence has been piling up since 2021. Google's engineers described "mercurial cores" — individual CPU cores that intermittently compute wrong results due to manufacturing defects that escaped every test. A mercurial core might be correct 99.99% of the time, failing only when a particular instruction meets particular operands at a particular temperature. Google traced real damage to them: corrupted encryption keys, broken database indexes, garbage collection deleting live data.

Meta ran the systematic version of the study across more than a million production CPUs and confirmed the shape of the problem: silent data corruption is not a black-swan event but a steady-state property of large fleets, occurring at roughly one fault per thousand devices. Compare that to the historical soft-error baseline of about one per million, and you understand why hyperscalers now run continuous in-production screening.

Three forces made this worse, and all three are accelerating:

• Silicon density. Smaller transistors mean more marginal ones, and more defects that slip past test coverage.
• Voltage and frequency scaling. Chips run closer to their electrical margins to hit efficiency targets, so a marginally defective circuit has less headroom before it misbehaves.
• Accelerators. GPUs pack enormous numbers of arithmetic units with less per-unit verification than CPU cores get, and AI workloads hammer exactly those units around the clock at high sustained temperatures.

The punchline for AI teams: the hardware you run on lies at a low but nonzero rate, and you bought a lot of it.

Determinism Was the Immune System

It's worth appreciating how good the old defenses were, because their loss is the whole story.

Classic distributed systems assume software is deterministic even when hardware isn't trustworthy. That single assumption powers every layer of corruption defense: checksums verify that bytes survived a journey; write-then-read-back verifies storage; running the same computation on two machines and comparing outputs catches a lying core with near certainty; replaying a suspect job against a golden output turns "the numbers look weird" into a binary verdict. Meta's fleet tooling is built on this — targeted micro-benchmarks with known-correct answers, run either during maintenance windows or sliced into milliseconds-long tests between production work. Google fuzzes CPUs continuously in production, comparing real silicon against the architecturally-defined result.

Every one of these techniques reduces to the same primitive: compute the thing twice, compare exactly. Exact comparison is what turns silent corruption into loud corruption.

Now look at LLM inference. Sampling is intentionally random. Floating-point reductions aren't associative, so results legitimately vary with execution order. And the sharpest finding of recent years: even at temperature zero, production inference isn't deterministic, because reduction kernels produce different numerics depending on batch size, and batch composition depends on whoever else's requests happened to arrive that millisecond. One team ran the same prompt 1,000 times through a popular open model at temperature zero and got 80 distinct completions — on healthy hardware. Bitwise comparison, the primitive underneath fifty years of corruption defense, returns "mismatch" all day on a perfectly good fleet.

So the alarm that used to fire on lying hardware now fires constantly for benign reasons, which is the same as never firing at all.

What a Lying GPU Looks Like in an LLM

The intuition most engineers carry — "a bit flip will produce a NaN or garbage tokens, and we'd notice" — turns out to be mostly wrong. Fault-injection studies of permanent GPU defects in LLM workloads found that NaN and infinity values account for only about 1% of silent-corruption outcomes. The overwhelming majority of corrupted values stay numerically plausible: an activation that's off by a few percent, a logit that's slightly too high, an attention score that shifts probability mass toward the wrong token. Less than 40% of bit-flip events are even single-bit; real defects corrupt values in messier, patterned ways.

Trace what happens when a defective multiply-accumulate unit sits in the path of your inference server:

• A corrupted value in an early layer propagates through every subsequent layer — transformers have no internal error correction, and normalization layers helpfully smear the damage across the whole hidden state.
• A bit flip in the KV cache is worse: the corrupted key or value gets re-read at every subsequent decoding step, so one flip taints the entire remainder of the generation.
• The visible symptom is not gibberish. It's a slightly wrong answer, a hallucinated detail, a reasoning chain that goes off the rails at step four — outputs indistinguishable from the model's ordinary failure modes.

Loading…