The Latency Budget Nobody Allocated Across Your Agent's Hops

Your agent has a latency SLO. Someone put it in a doc: "responses under 8 seconds, p95." What nobody did was decide how those 8 seconds get spent. There is no line item for the retrieval call, no line item for the planning step, no line item for the third tool the model decided to invoke because it felt uncertain. The budget exists as a single number at the boundary and as nothing at all inside. So when a five-hop chain blows past 8 seconds, the on-call engineer stares at a trace and cannot answer the only question that matters: which hop did it?

This is the difference between a service that has a latency budget and a service that has a latency hope. A budget is allocated per component and enforced. A hope is measured at the door and prayed over. Most agent systems ship with a hope, because the hop structure is dynamic — the model decides how many tool calls to make — and it feels impossible to budget something you don't control. It isn't. You budget it exactly because you don't control it.

The reason this bites agents harder than it bit the microservice generation before them is that agent hops are both more variable and more numerous than the RPC calls we were used to. A model round-trip has a p95/p50 ratio that can reach 4–6x — LLM-bounded systems show the widest tails of anything in your stack, because an extra reflection round or one more tool call can double the wall-clock with no warning. Stack five of those in a chain and the tail doesn't add. It multiplies. That's the part the 8-second number hides, and it's where we'll start.

The tail is multiplicative, not additive

Engineers reason about latency additively because that's how the median behaves. If retrieval takes 200ms at p50 and generation takes 2s at p50, the chain is about 2.2s at p50. Clean, intuitive, and completely wrong at the tail.

At the tail, what you care about is the probability that at least one hop is slow. That's a compounding probability, and it goes the wrong direction fast. The canonical result here is Dean and Barroso's "tail at scale": if a request fans out to 100 servers and each has a 1% chance of a slow response, the chance that at least one is slow is about 63%. A "1-in-100 rare event" happens on nearly two out of every three requests. The p99 of a single node becomes the p50 of the aggregate.

Agents don't fan out to 100 servers, but they don't need to. Consider a modest five-hop chain — plan, retrieve, tool call, tool call, synthesize — where each hop independently hits its own slow threshold 5% of the time. The probability that all five stay fast is 0.95^5 ≈ 0.77. So 23% of your requests eat at least one tail event. Your carefully measured p50 of 2.2s describes a request that, most of the time, doesn't actually happen the way you think.

This is why "our average is fine" is a trap. The average is dominated by the fast path. The user experience is dominated by the tail, and the tail grows with every hop you add. Each new tool you give the agent, each retry, each self-correction loop, is another Bernoulli trial against your SLO. The more capable you make the agent, the worse its tail gets — unless you budget for it.

Every hop spends from a budget nobody wrote down

Here is the mental model that fixes this. Your end-to-end SLO is a bank account. Every hop is a withdrawal. Right now, nobody wrote the withdrawals down, so the account overdrafts and you find out at the door.

The fix is the same one SRE teams have used for RPC services for a decade: decompose the SLO into per-hop budgets. An 8-second p95 might break down as: 300ms for input guardrails and routing, 1.5s for retrieval, 4s for the primary generation, 1.5s for tool execution across all tool calls, and 700ms of slack for serialization, queueing, and network. The numbers are negotiable. The existence of the numbers is not.

Two things change the moment you write these down. First, you can enforce them. A hop that has a budget can have a timeout set to that budget — and a timeout is the only thing that converts an unbounded tail into a bounded one. Without per-hop budgets, your only timeout is the global one, which means one slow retrieval consumes the budget that generation needed, and generation gets killed for retrieval's sin.

Second, you can attribute violations. When the chain blows 8 seconds, you don't ask "why was it slow." You ask "which hop exceeded its budget," and the answer is a subtraction. This is the entire game: turning an unattributable aggregate into a set of attributable line items.

The hard case is the dynamic hop count — the model decides to make three tool calls instead of one. You budget this the same way you'd budget any variable cost: cap it. Give the category a budget (1.5s total for all tool execution) rather than budgeting each individual call. If the agent wants to make five tool calls inside a 1.5s envelope, it can, but the envelope doesn't grow just because the model got chatty. The variability moves inside a fixed boundary instead of leaking out to the user.

You can't budget what you can't see per-span

Per-hop budgets are useless if your trace is a single span that says "agent: 9.4s." You need span-level attribution — one span per hop, nested to reflect the call structure, each stamped with how long it took and what it was.

The good news is that this stopped being a bespoke problem. The OpenTelemetry GenAI semantic conventions now standardize exactly this shape: each LLM invocation, each tool call, and each retrieval step becomes a child span, with gen_ai.* attributes carrying the model, provider, operation name, and token counts. Tool executions get their own execute_tool spans — and those spans are precisely where latency outliers surface, because tool latency is the part of the stack you understand least and control least.

What span-level attribution buys you, concretely:

• Critical-path analysis. In a chain with any parallelism, total latency is the longest path, not the sum. Span nesting shows you which hops are actually on the critical path and which are hiding in the shadow of a slower sibling. Optimizing an off-path hop buys you nothing; the trace tells you which is which.
• Per-hop tail tracking. You want a latency histogram per hop type, not just end-to-end. The end-to-end p99 tells you that you have a tail. The per-span p99 tells you the tail lives in retrieval, not generation — which is the difference between a caching project and a model-swap project.
• Time-to-first-token vs. total. For streaming responses, the hop that matters to perceived latency is time-to-first-token, which lives inside the generation span. A trace that only records total generation time hides the fact that the model started streaming at 800ms and the user was happy long before the span closed.

If your instrumentation records one number at the boundary, every diagnosis is a guess. If it records one span per hop with durations attached, diagnosis is arithmetic. That is the whole return on the instrumentation investment.

Bounding the tail once you can see it

Attribution tells you where the time went. Bounding the tail is what you do next, and the techniques are borrowed wholesale from the microservice playbook — they just apply cleaner now that you have per-hop budgets to hang them on.

Timeboxes at every hop. Each hop's timeout equals its budget. When a hop exceeds it, you don't wait — you act. This is the single highest-leverage change, because it's the only mechanism that turns an open-ended tail into a bounded one. An untimed hop has no p100; it has a p100 of "however long the slowest thing that ever happened took."

Hedged requests on the hops that support them. For idempotent, read-heavy hops like retrieval, fire a second request after a short delay (say, at the p95 mark) and take whichever returns first. Dean and Barroso showed this collapses the tail dramatically for a small increase in total load, because the odds of both attempts hitting the tail are the product of two small numbers. It doesn't work for non-idempotent tool calls — you can't hedge a payment — but it's nearly free for retrieval and re-ranking.

Partial results when a non-critical hop expires. If an enrichment hop blows its budget, render without the enrichment rather than making the user wait. Weight your budget allocation toward the hops on the critical user journey and let the peripheral ones fail open. A degraded-but-fast answer beats a complete-but-late one for almost every interactive agent.

Speculative and parallel execution where the structure allows. If two tool calls don't depend on each other, the model shouldn't run them in series. Much of an agent's tail is self-inflicted sequencing — hops that could have overlapped but didn't because the orchestration ran them one at a time. The critical-path view from your spans tells you exactly which serial chains are candidates for parallelism.

Start by writing the numbers down

The reason agent latency feels unattributable is not that agents are mysterious. It's that nobody did the boring allocation work up front. The SLO was a single number at the boundary, the hops inside were unbudgeted and untimed, and the tail was left to compound in the dark.

Everything above is downstream of one act: writing the per-hop budgets down. Once each hop has a number, you can time it, attribute against it, and bound it. The multiplicative tail stops being a mystery and becomes a line item you can point at. The five-hop chain that blows p99 stops being "the agent was slow" and becomes "retrieval spent 2.1s against a 1.5s budget on 8% of requests, here's the hedge."

Do the unglamorous thing first. Open your worst trace, draw the spans, and write a budget next to each one. You will almost certainly discover that the hop everyone blamed is fine, and the hop nobody watches is the one eating your tail. That discovery is the entire point — and it's waiting in a trace you already have.