The p99 of Thought: When the Model Decides How Long Your Request Takes

Every latency playbook you own was written for systems where the work per request was roughly constant. A database lookup takes what it takes. An image resize scales with pixel count, which you know in advance. Even classic LLM completions had a knowable cost envelope: input tokens in, bounded output tokens out. Reasoning models quietly deleted that assumption. When the model decides at runtime how long to think — and it decides based on how hard the problem turns out to be — response time stops being a property of your infrastructure and becomes a property of the question.

The consequence shows up first in your percentiles. Teams running reasoning models in production report p99 latencies spiking three to five times above p50, not because a host got slow or a cache went cold, but because one request in a hundred happened to be genuinely difficult. Your autoscaler, your timeout policy, and your SLO dashboard were all tuned for a world where that spread meant something was broken. Now it means the system is working as designed — and every tool you have for managing the tail is pointed at the wrong cause.

Latency used to be an infrastructure property. Now it's a model decision

With fixed-cost inference, tail latency had infrastructure explanations: queueing, contention, garbage collection, a bad replica. The Google SRE canon — most famously the "Tail at Scale" line of work — is built on that premise. The request itself was innocent; the environment made it slow. So the remedies were environmental too: hedge to another replica, drain the slow host, tune the scheduler.

Reasoning models add a second, larger source of variance that lives inside the request. Chain-of-thought length is input-dependent and decided incrementally: the model emits thinking tokens until its own internal sense of confidence says stop. Trajectories full of self-reflection and uncertainty markers keep going; easy questions exit early. Research on efficient reasoning consistently finds that this dynamic-exit behavior produces heavy right tails in per-request compute — a small fraction of inputs consume an outsized share of tokens and wall-clock time.

The uncomfortable part is that difficulty is not observable before you run the request. You can estimate it — routers and difficulty classifiers exist for exactly this reason — but the ground truth only reveals itself as the model thinks. That means your latency distribution is no longer stationary in any useful sense. It shifts with your traffic mix: a product launch that attracts harder questions, a new user cohort that pastes in longer documents, an agent workflow that starts hitting edge cases. Nothing in your infrastructure changed, and your p99 doubled.

There's also a compounding effect worth naming: overthinking. The efficient-reasoning literature shows models frequently reach the correct answer early in the trace and then keep generating verification steps anyway. So the tail isn't even purely "hard problems take longer." Some of it is the model burning tokens on problems it already solved — pure latency and cost with zero quality return. Your p99 contains both legitimate difficulty and pathological rumination, and from the outside they look identical.

"Just raise the timeout" converts a latency SLO into a cost incident

The reflexive fix, once reasoning requests start tripping your 30-second timeout, is to raise the timeout. This feels safe. It is not, and the reason is the billing model.

While a reasoning model thinks, you are paying for every token it emits — thinking tokens are metered like output tokens, and on frontier reasoning models output pricing runs several times input pricing. A timeout is therefore not just a latency bound; it is the only hard cost bound on a single request. Raise it from 30 seconds to 180, and each runaway request can now cost six times as much before your system gives up on it. Multiply by a retry policy that fires the same hard prompt at the same model — which will think just as long the second time, because the difficulty didn't go anywhere — and you've built a machine that responds to hard questions by paying for them repeatedly.

This is how a latency SLO quietly becomes a cost incident. The sequence is almost always the same:

• Tail requests start exceeding the timeout, and users see errors.
• The timeout gets raised so the errors go away.
• Retries and hedges are layered on to protect the remaining tail.
• The invoice arrives, and finance asks why token spend tripled while traffic grew 20 percent.

Autoscaling has the same blind spot. Most autoscalers key on request rate or GPU utilization, both of which assume requests are roughly interchangeable units of work. When per-request compute varies by an order of magnitude, request rate stops predicting load. Ten hard requests can saturate capacity that comfortably served a hundred easy ones. Newer serving-layer research addresses this with work-based signals — scaling on token velocity (the rate of tokens flowing through prefill and decode) rather than request count — but if your scaling policy predates reasoning models, it is almost certainly measuring the wrong thing and reacting after the queue has already built.

Hedging works — if you hedge against a different model, not a replica

The classic tail-at-scale remedy is the hedged request: if the primary hasn't answered by the p95 expected latency, fire a backup to another replica and take whichever finishes first. In Google's famous BigTable benchmark, this cut p99.9 from 1,800ms to 74ms for about 2 percent extra load. It worked because the slowness lived in the environment — a different replica wouldn't share it.

Reasoning latency breaks that assumption. If the request is slow because the problem is hard, an identical backup on an identical model thinks just as long. You pay twice and wait the same. Hedging survives, but the second request has to differ on the axis that actually causes the delay: reasoning effort.

Loading…