Capacity Planning for Agents: Why Concurrency, Not QPS, Is Your Real Unit

The first time an agent product falls over in production, the on-call engineer usually pulls up the request-rate graph and finds it boringly flat. Forty requests per second, same as yesterday. No spike, no surge, no obvious cause. Meanwhile the system is on fire: timeouts everywhere, the provider returning 429s, the queue depth climbing in a straight line toward the moon. The graph that should explain the outage looks completely healthy, because the graph is measuring the wrong thing.

QPS — queries per second — is the unit we inherited from a decade of stateless web services. A request comes in, you do tens of milliseconds of work, you send a response, the request is gone. Under that model, request rate and load are nearly the same number, and capacity planning is mostly "how many requests per second can one box handle, multiply by boxes." That mental model is so ingrained that we reach for it automatically. For agents, it quietly lies to you.

The reason it lies is that an agent request doesn't leave. A single user message kicks off a loop: the model reasons, calls a tool, waits for the tool, reads the result, reasons again, calls another tool, and around it goes for thirty seconds or three minutes or — when something goes wrong — until a timeout cuts it off. During that entire window the request is in flight. It is holding a slot, an LLM connection, a token budget, a chunk of context-window memory, and quite possibly a downstream database connection on every tool call. Ten requests per second sounds trivial. Ten requests per second that each live for two minutes is a very different animal.

The number you actually have to size for is concurrency

There is a piece of math that makes this precise, and it is old enough that nobody who serves traffic for a living should be surprised by it. Little's Law: L = λ × W. The average number of items in a system (L) equals the arrival rate (λ) times the average time each item spends in the system (W). It holds for any stable system regardless of the arrival distribution, which is what makes it such a reliable planning tool.

Plug agent numbers in. Suppose requests arrive at 10 per second, and the average agent run — reasoning plus all its tool calls, end to end — takes 90 seconds. Then:

L = 10 req/s × 90 s = 900 concurrent runs in flight at all times.

Nine hundred. That is the number you have to provision for: nine hundred simultaneous reasoning loops, nine hundred live LLM streams, up to nine hundred concurrent tool invocations hitting your downstream services at any given instant. The 10-per-second figure told you almost nothing useful. The 90-second residency time is what turned it into a load number, and the multiplier is brutal precisely because agent latency is measured in tens of seconds, not the 200 milliseconds a web request used to take.

This is the single most important reframe: latency is a multiplier on concurrency, not just a user-experience metric. In a web service, a latency spike is a UX problem. In an agent system, a latency spike is a capacity problem. If your average run time doubles — because a tool got slow, because the model started thinking harder, because a retry loop kicked in — your concurrent load doubles with it, even though not a single additional request arrived. The arrival graph stays flat while the system silently needs twice the capacity. That is exactly the outage where the QPS dashboard looks fine.

Fan-out makes the real number bigger than the request count

Concurrency at the request level is only the first layer. Modern agents fan out. One user request spawns sub-agents, runs tool calls in parallel, dispatches a batch of retrieval queries, maybe forks into a planner and several workers. A single inbound request can become dozens or hundreds of downstream calls before it resolves.

So the concurrency that matters at your provider, your vector database, and your internal APIs is not 900 — it is 900 times the fan-out factor at each tier. If each in-flight run holds, on average, three concurrent tool calls, your downstream services are seeing 2,700 simultaneous callers sourced from 10 requests per second of human input. Capacity planning that stops at the front door undercounts the real load by exactly this multiplier, and the multiplier is invisible on any request-rate graph.

This is where the failure mode turns vicious. Fan-out interacts with latency in a feedback loop. Downstream gets saturated, downstream latency rises, which raises the residency time W of every run holding a call there, which by Little's Law raises concurrency L, which means more runs piling more calls onto the already-saturated downstream. The system doesn't degrade gracefully along a smooth curve; it sits fine until it crosses the saturation point and then collapses, because each unit of extra latency recruits more concurrent load, which produces more latency. You don't get a warning slope. You get a cliff.

Why the inference layer punishes you for ignoring this

If you self-host inference, concurrency isn't just your accounting unit — it is the physical knob the serving stack turns, and it behaves nonlinearly. As you push more concurrent requests through a GPU, system-wide throughput (total tokens per second across all users) rises, because batching keeps the hardware busy. But per-user throughput falls: each individual user's tokens-per-second is roughly one over the inter-token latency, and inter-token latency climbs as the batch grows and requests contend for compute and memory bandwidth.

There is a saturation point. Below it, adding concurrency buys you throughput at a tolerable latency cost. Above it, system throughput plateaus while per-user latency keeps climbing — you're adding load and getting nothing back but slower responses, which (again) means longer residency, which means even more concurrency. The whole serving literature frames this as a trilemma: you cannot independently maximize throughput, minimize latency, and minimize cost. Pushing one corner deforms the other two. A capacity plan denominated in QPS has no vocabulary for any of this, because the thing that's saturating — the in-flight batch — never appears in a request-per-second number.

Even if you don't self-host, you inherit a version of the same physics through your provider's rate limits, which are increasingly denominated in concurrent requests and tokens-per-minute rather than raw QPS — because the provider, sizing their own fleet, already knows that concurrency is the unit that maps to GPUs.

Planning and operating in the right unit

The fix isn't exotic; it's a change of denominator applied consistently from estimation through runtime. A few practices follow directly from the math:

• Size with Little's Law, per boundary. For each tier — front-door runs, LLM calls, each downstream service — estimate L = λ × W using that tier's own arrival rate and residency time. Provision concurrency slots to cover L plus headroom for variance, not the long-run average alone. The minimum is a floor, not a target.
• Measure W obsessively, and alert on it as a capacity signal. Track end-to-end run duration and the residency time inside each tool. A rising W is an early warning that your concurrent load is climbing even while QPS is flat. Treat a latency regression as a capacity incident, because that's what it is.
• Make concurrency an explicit, bounded resource. Put a real cap on concurrent runs and concurrent tool calls — a semaphore, a bounded pool, a fixed-size queue — sized to what your downstreams can actually absorb at their service rate. Unbounded concurrency isn't "fast," it's just an outage scheduled for whenever traffic or latency ticks up.
• Apply backpressure instead of letting the queue absorb everything. When in-flight count hits the cap, make new work wait or shed it deliberately. An unbounded queue feels safer than a rejection, but it just converts a fast failure into a slow, total one — the queue grows, residency grows, and you crash with a million-item backlog instead of a clean 429.
• Budget the fan-out. Cap spawn depth and parallel tool calls per run. The difference between fan-out of 3 and fan-out of 30 is a 10× change in downstream concurrency from the identical inbound rate, and a single misconfigured loop can run that bill up before anyone looks at a graph.
• Capacity-test in concurrency, not QPS. When you load-test, fix the number of concurrent in-flight runs and find where W and error rates turn the corner. A test that ramps requests-per-second against an agent backend is measuring a quantity that doesn't govern the failure.

The throughline is a single sentence worth pinning above the dashboard: in an agent system, you are not sizing for how often requests arrive — you are sizing for how many are alive at once. QPS is the rate things enter. Concurrency is how many are simultaneously holding your resources, and because agent runs are long and fan out, that number is larger, more volatile, and far more dangerous than the request rate that's been the default unit of capacity planning for fifteen years. Switch the denominator, and the outage that came out of a clear blue sky turns back into a number you could have seen coming.