The p99 Is a Product Decision, Not an Infra One

There is a ritual that plays out on almost every team shipping an AI feature. Someone runs a load test, watches the p99 latency climb past two seconds, and files a ticket: "make it faster." The ticket lands on the infra team. They tune batch sizes, add GPUs, argue about the scheduler, and eventually claw the number down to 1.4 seconds. Everyone nods. The p99 is "handled."

The whole exercise is built on a false premise. The premise is that the latency target is a fact of the system — a physical constant the infra team discovers and then optimizes toward. It is not. The target is a choice, and it is a product choice. What counts as "fast enough" depends entirely on what the interface does while the user waits, and the interface is not the infra team's to design.

The tell is that two features with identical backend latency can have wildly different experiences. One streams tokens the instant the model starts generating; the user is reading by the half-second mark and never notices the response takes four seconds to finish. The other blocks on a spinner until the full answer is ready, then dumps it all at once. Same p99. One feels instant, the other feels broken. If the number were really an infra property, that wouldn't be possible.

The number you're optimizing is the wrong number

The first mistake is treating "latency" as a single scalar. For anything a human waits on, the perceptually decisive metric is time to first token (TTFT) — how long the screen stays blank before anything shows up. Total generation time barely registers by comparison.

The research is old and consistent. IBM's Walter Doherty found in 1982 that when system response dropped below 400 milliseconds, productivity and satisfaction jumped discontinuously; people entered a flow state and stopped waiting on the machine. Jakob Nielsen's thresholds from 1993 say the same thing: under 100ms feels instantaneous, up to about a second keeps the user's flow of thought intact, and past ten seconds you've lost them unless you're showing explicit progress. Human visual reaction time sits around 200ms, which is why chat interfaces that get first token under that mark feel genuinely snappy.

None of these thresholds care about your total latency. They care about the gap before feedback. A response that starts in 400ms and streams over four seconds beats a response that starts in two seconds and completes instantly — even though the second one has half the total latency. Optimizing end-to-end time while ignoring TTFT is optimizing the number the user doesn't feel.

There's a second metric hiding underneath: inter-token latency (ITL), the rhythm of the stream once it starts. A smooth 25ms-per-token cadence reads like natural typing. The same average with jitter — spiking to 200ms every ten tokens — produces visible stuttering that reads as the system struggling. Users perceive variance, not just the mean. Your p99 can look fine and the experience can still feel choppy because the distribution of inter-token gaps is ragged.

So before anyone files a "make it faster" ticket, the real question is: faster at what? TTFT, ITL, and total time are three different budgets, and they trade against each other. You can lower TTFT by starting to stream sooner even if total time gets slightly worse. That is almost always the right trade, and it is invisible if you only track one aggregate.

What the interface can hide, the backend doesn't have to deliver

Here is the reframe that moves the p99 out of the infra queue and into the design review: every millisecond the interface can hide is a millisecond the backend doesn't have to buy.

Front-end teams have known this for a decade under a different name — perceived performance. The toolkit is well-worn:

• Streaming is the single biggest lever. Showing tokens as they generate improves perceived responsiveness by 10–100x over a spinner, because the user starts reading within the TTFT window instead of waiting for the whole response. It costs you nothing in total latency; it just moves the user's clock forward.
• Optimistic UI updates the interface the instant the user acts, assuming success, and reconciles only if something fails. Twitter's like button animates before the network request resolves. The ~500ms of animation buys real time for the call to complete, and the interaction feels instantaneous because feedback was decoupled from the network.
• Skeleton screens mimic the shape of incoming content instead of showing a spinner. They set the expectation that content is arriving and measurably reduce perceived wait and bounce rates.
• A "thinking…" affordance reframes a slow answer as a careful one. The same three seconds reads as "the model is reasoning" instead of "the app is stuck" — the delay becomes evidence of quality rather than a fault.
• Pre-emptive loading on hover or focus starts the work before the user has finished deciding to ask for it, so the visible latency is whatever's left after the head start.

Loading…