48 posts tagged with "latency"

Your agent has a latency SLO. Someone put it in a doc: "responses under 8 seconds, p95." What nobody did was decide how those 8 seconds get spent. There is no line item for the retrieval call, no line item for the planning step, no line item for the third tool the model decided to invoke because it felt uncertain. The budget exists as a single number at the boundary and as nothing at all inside. So when a five-hop chain blows past 8 seconds, the on-call engineer stares at a trace and cannot answer the only question that matters: which hop did it?

This is the difference between a service that has a latency budget and a service that has a latency hope. A budget is allocated per component and enforced. A hope is measured at the door and prayed over. Most agent systems ship with a hope, because the hop structure is dynamic — the model decides how many tool calls to make — and it feels impossible to budget something you don't control. It isn't. You budget it exactly because you don't control it.

There is a ritual that plays out on almost every team shipping an AI feature. Someone runs a load test, watches the p99 latency climb past two seconds, and files a ticket: "make it faster." The ticket lands on the infra team. They tune batch sizes, add GPUs, argue about the scheduler, and eventually claw the number down to 1.4 seconds. Everyone nods. The p99 is "handled."

The whole exercise is built on a false premise. The premise is that the latency target is a fact of the system — a physical constant the infra team discovers and then optimizes toward. It is not. The target is a choice, and it is a product choice. What counts as "fast enough" depends entirely on what the interface does while the user waits, and the interface is not the infra team's to design.

Someone in a security review asks, "What's stopping a user from jailbreaking this into leaking the system prompt?" So you add an input classifier. Then someone asks, "What if the model generates something toxic?" So you add an output classifier. Then legal wants PII redaction, and the RAG team wants a groundedness check, and now every user message that used to take one model call takes four. Your p95 latency doubled, your inference bill went up 40%, and the demo that felt instant now has a visible pause before anything appears on screen.

That is the guardrail tax, and almost nobody budgets for it. The reflex to "add a guardrail" feels free because each individual check is cheap and obviously good. But guardrails don't compose for free — they stack in the critical path, each one a serial hop that adds latency, tokens, and a new dependency that can be down, rate-limited, or simply wrong.

The honest version of this conversation starts by admitting that a guardrail is not a feature flag. It is a second inference system bolted onto your first one, with its own failure modes and its own bill. Here's how to think about what it actually costs, and how to buy protection without paying retail for it.

Every latency playbook you own was written for systems where the work per request was roughly constant. A database lookup takes what it takes. An image resize scales with pixel count, which you know in advance. Even classic LLM completions had a knowable cost envelope: input tokens in, bounded output tokens out. Reasoning models quietly deleted that assumption. When the model decides at runtime how long to think — and it decides based on how hard the problem turns out to be — response time stops being a property of your infrastructure and becomes a property of the question.

The consequence shows up first in your percentiles. Teams running reasoning models in production report p99 latencies spiking three to five times above p50, not because a host got slow or a cache went cold, but because one request in a hundred happened to be genuinely difficult. Your autoscaler, your timeout policy, and your SLO dashboard were all tuned for a world where that spread meant something was broken. Now it means the system is working as designed — and every tool you have for managing the tail is pointed at the wrong cause.

Here is a number that should bother you more than it does: the median network round trip to a cloud LLM endpoint costs you somewhere between 48 and 800 milliseconds before the model emits a single token. That is not inference time. That is not queue time. That is the price of leaving the building — DNS, TLS, the haul to the nearest region, the haul back. For a single chat completion, you never notice it. For an agent that makes nine sequential tool calls to answer one question, you just shipped five seconds of pure transit latency, and your users feel every bit of it.

Most teams treat the round trip as a fixed cost of doing business, something you optimize around with streaming and spinners. But for a growing class of workloads, the round trip is the bug. The model you need is small enough to run where the data already lives, the task is repetitive enough that the marginal cloud cost compounds into real money, and the data is sensitive enough that shipping it to a third party is a liability you keep quietly accepting. On-device and edge inference is the option most engineering teams dismiss reflexively — "the local models aren't good enough" — and the dismissal is increasingly wrong.

The first thing a team does when its inference endpoint starts missing latency targets is buy more GPUs. The second thing it does, a month later, is notice that p99 latency barely moved while the bill doubled. The cards are sitting at 40% utilization and the tail is still ugly. Somebody adds an autoscaler. The autoscaler thrashes. Now there are more cards, more cost, and the same complaints from the same users.

The mistake is treating slow inference as a capacity shortage. It almost never is. What you have is a queueing problem wearing a capacity problem's clothes. Tail latency on an LLM endpoint is governed by how requests of wildly different sizes share a fixed pool of compute over time — which is the textbook definition of a scheduling problem, not a provisioning one. Until you understand the queue discipline your serving stack is actually running, every GPU you add is just a more expensive way to be slow.

The product team set the SLO for the new AI summarizer at 200ms TTFB because that is what the rest of the product hits at p50. Nobody on the call asked where the 200ms came from. It came from a decade of static assets and JSON responses served out of a CDN edge cache with an 85% hit rate, where most requests never reached origin and the ones that did were small. The summarizer is per-user, generated fresh each call, and travels edge → origin → model provider on every request. The SLO was structurally unmeetable on day one. The team discovered this in week six, after the dashboard had been red the whole time.

This is a recurring pattern in AI feature launches. The latency bar an organization built on top of one set of physics gets inherited by a feature with completely different physics, and the gap between the inherited target and the achievable floor becomes a months-long mitigation project instead of a Day 0 design constraint. The numbers do not care that the SLO was negotiated with a customer in good faith.

Your fine-tuned variant serves a few hundred requests per minute on a steady weekday, and the p99 latency dashboard is mostly flat. Then, at 03:14 local time on a Tuesday, p99 spikes from 800ms to 4.6 seconds for a single request, then settles back. The next night, it happens again, roughly the same shape, roughly the same hour. You file a ticket against the provider asking about the spike. The response is correct and unhelpful: their dashboard shows nothing anomalous on their side, no rate limits, no incidents, your token usage at the moment of the spike was unremarkable. The 4.6 seconds happened. The bill does not reflect it.

That gap — between a latency event a user clearly experiences and a bill that registers nothing — is the shape of the fine-tune cold start tax. It is not a bug in your code. It is not a regression on the provider's side. It is the seam where two billing models meet: the provider charges you for active inference time on the adapter, and the cost of loading the adapter into a serving slot is hidden inside the provider's infrastructure layer, where it shows up as your latency but their cost. If your traffic shape ever falls below the provider's keep-warm threshold, you pay for the round trip in p99 every time it climbs back.

The offline eval was unambiguous. After bolting a cross-encoder on top of the top-50 from vector search, nDCG@5 went up four points. The team shipped it on a Tuesday. By Thursday, p99 retrieval latency had crossed the SLO by 700 milliseconds, and customer success was forwarding screenshots of empty results pages that the old pipeline would have populated. The graph that mattered — user-perceived answer quality — was down. The reranker was a regression that the team had branded as an improvement, and the eval rubric was the thing that hid the regression in plain sight.

This is one of the most common failure modes in production retrieval, and it is rarely described as what it actually is: an evaluation bug. The reranker did what it was advertised to do. It reordered the top-50 with finer-grained precision. The problem is that the metric used to justify it — offline nDCG, computed at infinite budget, against the full reranked list — describes a world the production system does not live in. In production, the answer that ships is not the best-scored reranked list. It is whatever the system can return before the request deadline. And once you write the metric that way, the reranker's contribution is no longer a four-point lift. It is a curve.

The product spec says the user hears a response within 600 ms of finishing their sentence. The LLM provider's dashboard reports time-to-first-token at 280 ms. You are comfortably inside SLO on every chart you check. The user still complains the agent is laggy, and when you finally sit on a call yourself, there is a noticeable pause before the voice comes back — somewhere north of 600 ms, every time. The dashboard is not lying. It is measuring a number that does not include the TTS pipeline, the audio transport, or the jitter buffer on the receiving end. The 350 ms gap between the last token streamed and the first audio frame is real, it just is not on the LLM team's chart.

The bug is not in the model. The bug is in the SLO. It was defined at the wrong layer of the stack. The provider's egress is not the user's ear, and any latency contract that pretends otherwise will look healthy in production while the product feels broken.

A team I worked with last quarter shipped a reasoning-tier upgrade on a Tuesday and started getting support tickets on Wednesday. Users were saying the assistant felt "broken," "frozen," "hung." The on-call engineer pulled up the latency dashboard and found nothing unusual. p99 first-token latency was 612 ms — comfortably under the 800 ms SLO that the team had spent a quarter establishing. The dashboard was green. The phone was ringing.

The bug turned out to be a single instrumentation decision made fourteen months earlier, before reasoning models existed in production. The metric labeled "first token" measured the timestamp on the first chunk emitted by the provider. After the upgrade, the first chunk was a reasoning token — invisible to the user, never rendered, but counted as "first" by the SLO. The model was emitting four to seven seconds of internal thoughts before the first user-visible character streamed. Every chart stayed green. Every user waited in the dark.

This is not a story about a bad metric. The metric was correct for the model it was designed against. It is a story about what happens when the boundary you instrumented stops being the boundary your users feel — and how dangerously easy it is to ship that drift without noticing.

The launch metrics looked clean. Answer quality up, citation rate up, the eval suite green. The team that replaced the old keyword search with an agent-backed retriever shipped, took the win, and moved on. Six weeks later somebody noticed the weekly active number on that surface had drifted down twelve percent and nobody could find the regression. There was no regression. The agent worked. The users left because the box that used to answer in two hundred milliseconds now took four seconds, and nothing in the launch retro had a budget for that.

This is the latency-budget transfer problem, and almost nobody draws the org chart that catches it. A search box is not just a function call. It is a thirty-year contract with the user's nervous system: type, see results, scan, click. The 200-millisecond response is not a performance metric on a dashboard somewhere — it is the reason the user's attention is still on the screen when the results arrive. When the team underneath the box replaces a keyword index with an agent loop, the function-call surface looks identical and the SLA on the new call lives in a completely different regime. The latency budget moved from the team that owned the index to the team that owns the agent, and from the team that owns the agent to the user, and the only one who showed up to the meeting was the user.