The Thundering Herd Behind Your 429s: Rate Limits Are a Distributed Systems Problem

Pull up your request logs from the last time you hit sustained 429s. You will probably find something odd: the errors don't arrive as a steady stream. They arrive in waves — a burst of 429s, a quiet gap, a bigger burst, another gap. The provider's quota didn't change between waves. Your traffic didn't spike. What you are looking at is your own retry logic, synchronized against itself. Every client that failed at second zero computed the same backoff delay, slept the same duration, and woke up at the same instant to fail together again.

This is the thundering herd, and the punchline is that the standard fix — exponential backoff — does not prevent it. Deterministic exponential backoff organizes the herd. It takes a crowd of clients that failed at roughly the same moment and marches them forward in lockstep: everyone retries at 1 second, then everyone at 2, then everyone at 4. The load spikes get farther apart, but each spike is just as tall as the first. If the spike is what triggered your rate limit, you have built a metronome that re-triggers it forever.

Most teams discover this the hard way, because rate-limit handling is treated as a snippet you copy from the provider's docs rather than what it actually is: a distributed systems problem. Dozens or hundreds of your processes — API workers, batch jobs, agent loops — are competing for a shared, externally enforced resource with no coordinator. The moment you take that framing seriously, thirty years of distributed systems results apply directly, and they say three things: randomize your retries, cap your total retry volume, and eventually you will need to build a scheduler. Let's take those in order.

Why Exponential Backoff Synchronizes Your Clients

The failure mode is easiest to see with a concrete timeline. Suppose 100 workers each fire a request in the same 200-millisecond window and your provider throttles all of them. Every worker runs the same retry code: wait base × 2^attempt, retry. With a one-second base, all 100 workers sleep one second and wake up inside the same 200-millisecond window. If the provider's per-minute budget can absorb, say, 60 of them, the other 40 get throttled again — and now those 40 are perfectly synchronized, because they all failed at the same wall-clock instant. By the third round, the survivors of each wave form tighter and tighter clusters. The backoff schedule is deterministic, the trigger was shared, so the retries are correlated by construction.

AWS demonstrated this with a simulation that has since become the canonical reference: clients contending for a shared resource under optimistic concurrency, retrying with pure exponential backoff. The result was that contention barely dropped — the retries still arrived in clusters, just spaced farther apart. The exponential schedule spreads the waves out in time; it does nothing to spread the clients out within a wave.

There's a subtle aggravator in LLM workloads specifically: the failure trigger is more shared than in most systems. When a traditional service degrades, clients fail at slightly different times as their individual requests time out. When a provider's token-per-minute bucket empties, everyone who arrives after that instant gets a 429 simultaneously and instantly. The synchronization is sharper, so the herd is tighter. Agent frameworks make it worse still — a multi-agent pipeline that fans out five parallel tool calls per step is already a small coordinated burst before any retry logic runs.

Jitter Is Not a Nicety — It Is the Actual Fix

The repair is randomness. Instead of sleeping exactly base × 2^attempt, each client sleeps a random duration drawn from a window that grows exponentially. The standard variants, in ascending order of how aggressively they decorrelate:

• Equal jitter: sleep half the exponential delay, plus a random amount up to the other half. Keeps a guaranteed minimum wait, randomizes the rest.
• Full jitter: sleep a uniformly random duration between zero and the full exponential delay. The entire wait is randomized.
• Decorrelated jitter: sleep a random duration between the base and three times your previous sleep, capped. Each client's schedule drifts independently of both the attempt count and every other client.

In the AWS simulations, the no-jitter variant performed so much worse than everything else that it had to be dropped from the comparison charts. Among the jittered variants, full jitter won on the combined measure of server load and time-to-completion, and it remains the default recommendation — including in most official LLM provider SDKs, which is worth checking rather than assuming: several popular clients ship with jittered backoff enabled, and teams then layer their own naive retry loop on top, reintroducing the synchronization the SDK had already solved.

Two practical notes that the blog-post version of this advice usually omits. First, jitter belongs on the first attempt too, not just retries. If a cron tick or a queue drain releases 500 jobs at the top of the minute, they are a synchronized herd before anything has failed; a few hundred milliseconds of random start delay dissolves it.

Second, when the provider sends a retry-after header — and the major LLM APIs do — honor it, but jitter around it. If 80 clients all receive retry-after: 30 at the same moment, sleeping exactly 30 seconds recreates the lockstep wave with the provider's own blessing. Treat the header as a floor, add a random spread on top.

Backoff Delays the Herd; Budgets Kill It

Here is the uncomfortable result from production incident reports: even correctly jittered exponential backoff only postpones load amplification during a sustained outage. It does not bound it.

The arithmetic is simple and brutal. If your client retries each failure twice, then during an incident where most requests fail, every logical request becomes three physical requests — 3× amplification. If service A retries calls to B, and B retries calls to C, the amplifications multiply: two layers of two retries is 9× load at the bottom of the stack.

Loading…