The Prompt-Cache Cliff: How One System-Prompt Edit Re-Priced Your Whole Fleet

Nothing broke. That's the disorienting part. No deploy failed, no latency alarm fired, no error rate ticked up. Someone merged a one-line PR that appended a sentence to the system prompt — a new tool description, a policy reminder, a "today's date is" header — and the next morning the inference bill was three to five times higher. Traffic was flat. The model was the same. The code did exactly what it was supposed to do.

What changed is that the one line landed in the wrong place, and every cached prefix in your fleet invalidated at once. Your cache hit rate went from 90% to zero in a single request cycle, and every token that used to be nearly free started billing at full price. This is the prompt-cache cliff, and it's the most expensive failure mode in production LLM systems that nobody threat-models, because it doesn't look like a failure at all.

The reason it's invisible is that prompt caching is a discount you never explicitly turned on. Providers apply it automatically when they recognize a repeated prefix, so for months your system runs cheap and you never think about it. Then the discount silently evaporates and the "new" price is just the price without the thing you didn't know you were relying on. If you don't have an alarm on cache-hit-rate, the first signal you get is the invoice — and by then you've been paying the cliff price for a full billing cycle.

Caching is a prefix match, and that's the whole story

Every major provider's prompt cache works the same way underneath: it's an exact-prefix match on the rendered bytes of your request. The model processes your prompt token by token, and the expensive part — building the key-value attention state — can be reused if the beginning of a new request is byte-identical to one it already computed. The provider hashes your prompt up to certain checkpoints and, on a hit, loads that precomputed state instead of recomputing it.

The consequence follows directly: any change anywhere in the prefix invalidates everything after it. Not the changed token — everything downstream of it. The cache key is derived from the exact bytes up to each checkpoint, so a single differing byte at position N kills every checkpoint at position ≥ N. A trailing space, a reordered JSON key, a reformatted date, one extra sentence in the system prompt: all of them are total invalidations of the prefix that follows.

This is why the order of your prompt is a cost decision, not a stylistic one. Providers render the request in a fixed sequence — on Anthropic it's tools, then system prompt, then messages. Anything you put early sits in front of everything else. A timestamp interpolated into the system-prompt header doesn't just fail to cache itself; it makes the entire system prompt and every tool definition behind it uncacheable, because they now live downstream of a value that changes every request.

The "accreting system prompt" and prompt caching are on a collision course for exactly this reason. System prompts grow. Every incident spawns a new "always remember to…" line, every feature adds a tool, every edge case adds a caveat. Each of those edits sits at the front of the prefix. And the bigger the stable prefix you've built up, the more you have riding on it — which means the blast radius of touching it grows over time, not shrinks.

The economics of a write, and why the cliff hurts so much

To see why an invalidation is expensive rather than merely inconvenient, you have to look at the price structure. Providers charge three different rates for input tokens:

• Cache reads — tokens served from an existing cache entry. On Anthropic these cost about 0.1× the base input rate, a 90% discount. For Claude Opus 4.8, that's $0.50 per million tokens read versus $5.00 uncached.
• Cache writes — tokens written into the cache on a miss. On Anthropic these cost more than base input: 1.25× for the default 5-minute TTL, 2× for the 1-hour TTL.
• Uncached input — full price, no caching involved.

In steady state, most of your prompt tokens are cache reads at 0.1×, which is why a well-cached system is cheap. When someone edits the prefix, two things happen simultaneously. First, the next request on every cached path is now a full cache write — you pay the 1.25× premium to rebuild the entry. Second, until that rebuild happens, those tokens bill at full uncached rates. The steady-state rate was 0.1× of base; the cliff rate is 1× to 1.25× of base. That's the 3–5× jump, and it applies to every concurrent path at once because they all shared the prefix you just changed.

Loading…