GPU Scheduling Is a Queueing Problem, Not a Provisioning Problem

The first thing a team does when its inference endpoint starts missing latency targets is buy more GPUs. The second thing it does, a month later, is notice that p99 latency barely moved while the bill doubled. The cards are sitting at 40% utilization and the tail is still ugly. Somebody adds an autoscaler. The autoscaler thrashes. Now there are more cards, more cost, and the same complaints from the same users.

The mistake is treating slow inference as a capacity shortage. It almost never is. What you have is a queueing problem wearing a capacity problem's clothes. Tail latency on an LLM endpoint is governed by how requests of wildly different sizes share a fixed pool of compute over time — which is the textbook definition of a scheduling problem, not a provisioning one. Until you understand the queue discipline your serving stack is actually running, every GPU you add is just a more expensive way to be slow.

This is unintuitive because it contradicts how we reason about almost every other backend service. For a stateless web tier, latency really is mostly a capacity story: add replicas, spread load, done. LLM serving breaks that intuition for a specific reason — the unit of work is not uniform, its size is unknown in advance, and it holds expensive memory for its entire lifetime. Those three properties turn a fleet of fast accelerators into a system whose throughput is set by queue dynamics, and where the bottleneck moves around depending on what just arrived.

Why More Cards Doesn't Buy You Less Tail

Start with what a GPU is actually doing during inference. An LLM request has two phases with opposite resource profiles. Prefill ingests the prompt in a single dense forward pass where every token attends to every other token — this is compute-bound, limited by raw FLOPS. Decode then generates output tokens one at a time, and each step must reload the key-value cache for every prior token from memory — this is memory-bandwidth-bound. The GPU spends most of decode loading KV tensors, not computing.

A single card therefore cannot be "utilized" in any simple sense, because the two phases stress different parts of the chip. When you add a card, you add both kinds of capacity in a fixed ratio — but your workload's mix of prefill and decode shifts second to second. The result is that aggregate utilization stays low (because one resource is always idle relative to the other) while the bottleneck resource for whatever just arrived is saturated. You bought a balanced quantity of compute for an unbalanced, time-varying demand.

The deeper issue is head-of-line blocking. Suppose a request with a 32,000-token prompt lands in the middle of a batch that's busy decoding for fifty other users. That prefill is a large, compute-heavy unit of work, and while the GPU grinds through it, every decode step for those fifty users stalls. Their inter-token latency spikes. None of them did anything wrong; they're just behind a whale in the queue. Adding a GPU does nothing for them unless the whale lands on the other card — and with naive load balancing, it's a coin flip. You've doubled cost to improve the odds of avoiding a stall, which is not the same as fixing the stall.

This is why the utilization-versus-tail paradox is so common. The fleet looks underused on a dashboard precisely because the requests that matter are spending their time waiting, not running. Idle silicon and bad p99 are not a contradiction. They are the signature of a scheduling failure.

The Three Knobs That Actually Move Throughput

If provisioning isn't the lever, what is? Three queue-discipline mechanisms determine your real throughput, and all of them are about how work is admitted, ordered, and packed — not how much hardware exists.

Loading…