Retries Aren't Free: The FinOps Math of LLM Retry Policies

A team I talked to last quarter found a $4,200 line item on their inference invoice that nobody could explain. The dashboard showed normal traffic. The latency graphs were flat. The cause turned out to be a single agent stuck in a polite retry loop for six hours, replaying a 40k-token tool chain with exponential backoff that capped out at thirty seconds and then started over. The retry policy was lifted verbatim from an internal SRE handbook written in 2019 for a JSON-over-HTTP service. It worked perfectly. It worked perfectly for the wrong system.

This is the bill that does not show up in capacity-planning spreadsheets. The retry-policy patterns the industry standardized on for stateless REST APIs assume three things that LLM workloads quietly violate: failures are transient, the cost of one extra attempt is bounded, and a retry has a meaningful chance of succeeding. Each assumption was load-bearing. Each one is now wrong, and the variance the cost model never captured is sitting at the bottom of every monthly invoice.

The teams that have not rebuilt their retry policy for token economics are paying a hidden tax that scales with the difficulty of the queries they were already most worried about — the long ones, the agentic ones, the ones with deep tool chains. The retry budget that classical resilience engineering hands you back as a safety net is, in an LLM stack, the rope.

The Three Classical Assumptions That LLM Workloads Break

Retry policy as a discipline matured around stateless REST APIs and idempotent RPC. The mental model assumed: a request is a fixed-cost unit of work, the failure is usually a network blip or a transient backend hiccup, the retry costs roughly one extra unit, and the second attempt has independent odds of succeeding. The math worked because every variable in it had a stable distribution.

LLM calls violate all three. The cost of a request is not a fixed unit — it is a function of input tokens, output tokens, cache hit ratio, reasoning depth, and tool-call recursion, each of which can vary by an order of magnitude per call. The failure is rarely a network blip; it is more often a malformed JSON field, a refused completion, a tool that returned the wrong shape, or a reasoning loop that ran out of step budget. And the second attempt's success odds are not independent of the first — the model that produced a malformed schema once is highly correlated with producing one again, because the input it was responding to has not changed.

The cost formula matters here. A reasonable rough model of a per-request bill is roughly multiplicative: base_tokens × cache_multiplier × batch_multiplier × reasoning_multiplier × retry_multiplier. The retry term is not additive padding on top of a stable bill; it multiplies through the other terms, which is why a misconfigured retry can triple a monthly invoice rather than nudge it up 5%. The teams that budget retry as a flat 5–15% traffic uplift are anchoring on the wrong shape of cost — that figure is a fine starting heuristic for steady-state behavior, but it does not bound the tail.

Retries Are Most Expensive on the Inputs That Most Need Retrying

The hidden cruelty of LLM retry economics is that the cost amplification is highest on exactly the inputs you least want to amplify. A short, well-formed prompt with a one-shot completion fails rarely and replays cheaply. A long agentic prompt with eight tool calls, a 60k-token context window, and a deep reasoning chain fails more often — and a retry replays the whole chain at full token cost, including any cache misses introduced by the new attempt's slightly different prefix.

If your retry policy is unconditional ("on failure, retry up to N times"), the marginal token spend on retries is biased toward the most expensive requests in your distribution. The 5–15% headline traffic uplift becomes a 30–50% bill uplift on the long-tail inputs, and your unit economics quietly skew. The team that benchmarked cost-per-task on the median input is now serving the p99 input at an entirely different price, and the dashboard that rolls up daily averages does not surface the divergence until the invoice arrives at the end of the month.

The agentic case is sharper still. A failed step inside a tool-calling loop typically rewinds to the last consistent state and replays from there — which means retry cost is not the cost of one extra LLM call, it is the cost of the tool calls and reasoning steps that preceded it on the rewind. A team I know found that one of their failure modes was costing 4× its happy-path equivalent because the retry boundary was set at the agent loop and not at the failed sub-step. They had been thinking of retries as a tax on the LLM bill; the tax was actually on every upstream tool call the rewind invalidated.

Same Prompt, Same Failure: The Semantic Correlation Problem

The other broken assumption is statistical. Classical retry math assumes the second attempt has independent odds of succeeding — independent in the sense that the failure was caused by something outside the request itself (network, capacity, transient overload). For LLMs, the failure is often caused by the request itself: a prompt that confuses the model into producing malformed JSON on attempt one is highly likely to produce malformed JSON on attempt two, because nothing about the prompt changed. Empirical analyses of agent loops have found retry-resolves-it rates as low as around 20% for structural failures, with the rest of the budget burned producing different-but-still-failing outputs.

This is the failure mode that makes naive retry policy look the most rational while being the least effective. The first attempt fails. The retry runs. The retry fails differently — different enough that the stagnation detector does not catch it, but not different enough that the downstream parser accepts it. The agent burns through its retry budget producing variants of the same failure, and the cost shows up as compute spent without progress. A retry that does not change the input has no entitlement to a different outcome, and treating temperature noise as a retry strategy is mostly a way to convert determinism into a bill.

The split that matters is between transient failures (provider 503, rate-limit 429, network timeout — where a retry has independent odds) and semantic failures (malformed output, tool-shape mismatch, refusal — where a retry without input change is correlated with itself). Classical retry policy treats both as "failure, try again." LLM retry policy has to distinguish them, because the two have completely different cost-per-attempt math and completely different probability of resolution.

The Retry-Budget Pattern, Rebuilt for Tokens

The pattern that gives back the cost predictability classical retry policy assumed has four moving parts, and most teams have one or two of them but rarely all four.

Per-request and per-session token caps. Before any retry, set a hard ceiling on total tokens spent on this logical unit of work. The ceiling is enforced at the session boundary, not just the request boundary, so a runaway agent loop hits the cap regardless of how many sub-requests it has issued. This is the mechanical equivalent of a circuit breaker on cost. Pre-execution policy enforcement matters because the alternative — discovering the cost on the invoice — is the failure mode the pattern exists to prevent.

Fall-through to a cheaper model on retry. If the first attempt fails on a frontier model, the retry should not be a verbatim replay against the same model — it should fall through to a cheaper tier, or to a structurally different prompt. The economic reasoning is direct: a frontier-model retry costs nearly the same as the original call, and if the original failed for a non-transient reason, the retry has 20% odds of helping. A cheaper-model retry costs a fraction of the original and can absorb the cases where the failure was due to model overconfidence rather than capability ceiling. For provider-side outages, the fall-through is to a different vendor entirely, with a circuit breaker that takes the failed provider out of rotation for a cooldown period.

Semantic-failure classifier. The retry decision is not "did the call fail" but "did the call fail in a way a retry will help." A 429 or 503 says yes; a malformed JSON or a refusal says probably not, at least not without changing the input. The classifier is usually a few lines of code matching error types and output shapes, and its job is to decide whether to retry, modify-and-retry (with a corrected prompt that includes the validation error), or fail-fast. The modify-and-retry path — feeding the parser error back to the model — is where the empirical win is, because it actually changes the input the model is responding to.

Retry budget as a token bucket, not a count. The classical retry budget is "N retries per minute." The LLM retry budget is more useful as a token bucket: this caller may spend up to X retry-tokens per minute. A failed completion of a 40k-token call consumes more retry-budget than a failed completion of a 2k-token call, which is the right shape because the cost amplification is what the budget is trying to bound. When the bucket is empty, the caller falls through to a degraded path — cached response, simplified prompt, or human escalation.

What to Instrument So You Can See the Bill Coming

The reason retry-induced cost spikes show up as month-end invoice surprises rather than mid-month alerts is that the standard dashboards are denominated wrong. QPS and latency graphs aggregate failures into a single counter. The cost-amplifying retries are invisible because the dashboard shows "request rate" while the bill shows "tokens consumed" — and the relationship between the two is exactly the variable the retry storm is bending.

The instrumentation that makes this visible has a few non-negotiable shapes. Tag every run with an outcome state: accepted, rejected, abandoned, timeout, tool-error, retry-exhausted. Track the failure-cost share — the percentage of monthly token spend that landed on a non-accepted outcome — and alert when it exceeds a threshold. Break out cost per retry-attempt separately from cost per first-attempt, so a regression in retry rate is visible as a cost line and not just a counter. And capture per-session and per-tenant retry-token consumption, because the noisy-neighbor case where one tenant's retry storm drains the shared budget is otherwise invisible until customers complain about latency.

The deeper organizational shift is that retry policy is no longer something the platform team writes once and forgets. The cost amplification depends on the prompt distribution, the tool-call shape, and the model mix — all of which evolve with every product release. The teams that have made it through the worst incidents treat retry policy as a quarterly review surface, with the cost data on one side of the table and the reliability data on the other, looking for the cases where the two are pulling against each other.

The Realization Worth Holding Onto

The 2018 SRE playbook was right that retries are a resilience primitive — and it was right that the cost of getting them wrong was bounded. Both sentences still apply to most of the API surface a service exposes to its callers. The LLM call site is the exception, and the exception is large enough that a team applying the playbook unmodified will discover the boundary by being on the wrong side of it.

A retry against an LLM is not "the same call again." It is a probabilistic re-roll on a non-uniform cost distribution, with success odds that are correlated with the failure of the previous attempt and a price tag that scales with the difficulty of the input. The retry-budget pattern in this article is the minimum equipment to put bounds back on that distribution. The teams that ship it before they need it pay a small refactor cost. The teams that ship it after a $4,200 line item pay the line item plus the refactor.

Cost predictability used to be a property of the retry primitive. It is now a property of the retry policy you build on top of it.