The Postmortem With No Root Cause

The incident bridge is quiet in a way that means everyone is stuck. A support ticket says the agent told a customer their refund was approved when it wasn't. You have the full trace: the prompt, the retrieved account record, the tool calls, the model's reasoning, the final message. You replay it. The agent does the right thing. You replay it again. Right again. Nine times out of ten, the trace that produced the incident produces a correct answer instead. Someone on the call finally asks the question that the retro template can't handle: so what was the root cause?

There isn't one. Not in the sense the template means. The five-whys chain runs "the agent told the customer the wrong thing" → "because the model generated an approval" → "because it sampled a token sequence that asserted approval" → "because... that's what the probability distribution allowed." The last why bottoms out in a shrug. "The model sampled a bad token" is technically true and operationally useless. It names no fix, assigns no owner, and closes no gap. You can write it in the report, but everyone reading it knows you've documented a coincidence, not a cause.

This is not a new problem wearing new clothes. Incident review has always struggled with complex systems — Richard Cook's "How Complex Systems Fail" argued decades ago that overt failure requires multiple faults and that there is no isolated cause of an accident, only contributors. What LLM systems do is take that already-shaky assumption and detonate it. When the same input yields success and failure on different runs, the deterministic causal chain that root-cause analysis walks backward simply isn't there to walk. The retro template assumes a determinism the system does not have.

The Retro Template Is the Bug

Most incident processes inherit their shape from manufacturing and aviation, where a defect is a defect: the part was out of spec, the bolt was torqued wrong, and once you fix that specific thing the failure mode is closed. The five-whys method encodes this assumption directly. It follows a single string of cause and effect, deliberately narrowing toward one fixable thing at the bottom. That works beautifully when there is one fixable thing.

An LLM failure doesn't have that structure. The output that hurt you was drawn from a distribution, and the distribution assigned nonzero probability to the bad output given everything upstream — the prompt wording, the retrieved context, the tool result ordering, the system instructions, the sampling temperature. None of these individually "caused" the failure. Together they set a probability, and the sampler rolled the dice. Asking "why did it fail" is like asking why a specific coin flip came up tails. The honest answer is "because P(tails) was greater than zero," and that answer is correct and worthless at the same time.

The tell that you're fighting the template rather than the incident: your report has a root-cause field, someone has to fill it, and the only true thing to write is a sentence about sampling that no action item can follow from. When the honest answer to your form's mandatory field is a non-answer, the form is wrong, not the incident.

From "Why" to "How," From Cause to Probability

The escape hatch has two moves, and neither is new to incident practice — they were just never load-bearing before.

The first is to stop asking why and start asking how. John Allspaw's "Infinite Hows" reframing, born from blameless-postmortem culture, asks what conditions had to exist for the event to be possible rather than which single fault to blame. Applied to a stochastic system, this maps cleanly: instead of "why did the model approve the refund," ask "how did the system arrive in a state where approving the refund was a plausible next token?" That question has real answers. The retrieved record was ambiguous about refund status. The prompt never explicitly said the agent lacks approval authority. The tool that checks eligibility returned before the tool that checks authorization, so the model saw "eligible" without "not yet authorized." Each of these is a contributing factor you can actually change, and none of them requires the model to have been deterministic.

The second move is to stop talking about the incident as a single event and start talking about it as a rate. The failure you're reviewing is a sample from a distribution, so the honest unit of analysis is P(failure), not "the failure." This reframes the entire report. You are no longer explaining one bad output; you are explaining why the bad-output probability was high enough to fire in production, and what you changed to lower it. A report that says "we reduced P(hallucinated approval) from an estimated 4% to 0.3% on the eligibility-ambiguous slice" is honest, actionable, and verifiable. A report that says "we fixed the bug" is a lie about a system that has no bugs in the classical sense — only rates.

Reproduce at a Rate, Not at All

Loading…