Nothing broke. That's the disorienting part. No deploy failed, no latency alarm fired, no error rate ticked up. Someone merged a one-line PR that appended a sentence to the system prompt — a new tool description, a policy reminder, a "today's date is" header — and the next morning the inference bill was three to five times higher. Traffic was flat. The model was the same. The code did exactly what it was supposed to do.
What changed is that the one line landed in the wrong place, and every cached prefix in your fleet invalidated at once. Your cache hit rate went from 90% to zero in a single request cycle, and every token that used to be nearly free started billing at full price. This is the prompt-cache cliff, and it's the most expensive failure mode in production LLM systems that nobody threat-models, because it doesn't look like a failure at all.
The reason it's invisible is that prompt caching is a discount you never explicitly turned on. Providers apply it automatically when they recognize a repeated prefix, so for months your system runs cheap and you never think about it. Then the discount silently evaporates and the "new" price is just the price without the thing you didn't know you were relying on. If you don't have an alarm on cache-hit-rate, the first signal you get is the invoice — and by then you've been paying the cliff price for a full billing cycle.
Caching is a prefix match, and that's the whole story
Every major provider's prompt cache works the same way underneath: it's an exact-prefix match on the rendered bytes of your request. The model processes your prompt token by token, and the expensive part — building the key-value attention state — can be reused if the beginning of a new request is byte-identical to one it already computed. The provider hashes your prompt up to certain checkpoints and, on a hit, loads that precomputed state instead of recomputing it.
The consequence follows directly: any change anywhere in the prefix invalidates everything after it. Not the changed token — everything downstream of it. The cache key is derived from the exact bytes up to each checkpoint, so a single differing byte at position N kills every checkpoint at position ≥ N. A trailing space, a reordered JSON key, a reformatted date, one extra sentence in the system prompt: all of them are total invalidations of the prefix that follows.
This is why the order of your prompt is a cost decision, not a stylistic one. Providers render the request in a fixed sequence — on Anthropic it's tools, then system prompt, then messages. Anything you put early sits in front of everything else. A timestamp interpolated into the system-prompt header doesn't just fail to cache itself; it makes the entire system prompt and every tool definition behind it uncacheable, because they now live downstream of a value that changes every request.
The "accreting system prompt" and prompt caching are on a collision course for exactly this reason. System prompts grow. Every incident spawns a new "always remember to…" line, every feature adds a tool, every edge case adds a caveat. Each of those edits sits at the front of the prefix. And the bigger the stable prefix you've built up, the more you have riding on it — which means the blast radius of touching it grows over time, not shrinks.
The economics of a write, and why the cliff hurts so much
To see why an invalidation is expensive rather than merely inconvenient, you have to look at the price structure. Providers charge three different rates for input tokens:
Cache reads — tokens served from an existing cache entry. On Anthropic these cost about 0.1× the base input rate, a 90% discount. For Claude Opus 4.8, that's $0.50 per million tokens read versus $5.00 uncached.
Cache writes — tokens written into the cache on a miss. On Anthropic these cost more than base input: 1.25× for the default 5-minute TTL, 2× for the 1-hour TTL.
Uncached input — full price, no caching involved.
In steady state, most of your prompt tokens are cache reads at 0.1×, which is why a well-cached system is cheap. When someone edits the prefix, two things happen simultaneously. First, the next request on every cached path is now a full cache write — you pay the 1.25× premium to rebuild the entry. Second, until that rebuild happens, those tokens bill at full uncached rates. The steady-state rate was 0.1× of base; the cliff rate is 1× to 1.25× of base. That's the 3–5× jump, and it applies to every concurrent path at once because they all shared the prefix you just changed.
The break-even math makes the write premium concrete. With a 5-minute TTL, caching pays off after just two requests (1.25× write + 0.1× read = 1.35× versus 2× for two uncached calls). The 1-hour TTL doubles the write cost, so it needs at least three requests to break even, but it survives longer gaps in bursty traffic. Either way, the model assumes you reuse the prefix many times. An edit that forces a rebuild throws away all the amortization you'd banked and starts the meter over.
Other providers file the edges off but keep the shape. OpenAI's caching is fully automatic — it kicks in for prompts over 1,024 tokens, matches the longest previously-seen prefix in 128-token increments, and charges no write premium, just a 50% discount on cached reads. Google's Gemini offers both implicit caching (on by default, ~90% off, hashes the start of your request) and explicit caching (you create a named cache with a TTL, pay a small write fee, and then pay an hourly storage rent — $4.50 per million tokens per hour for Pro-tier models). The knobs differ; the invariant doesn't. Move content at the front of the prefix and you forfeit the discount behind it.
Architecting for the cliff: stable up top, volatile at the bottom
The defense is structural, and it's the same on every provider: order your prompt from most stable to least stable, and never let a volatile value sit in front of a stable one.
Concretely, the layout that caches well is: tool definitions first (they render at position zero and must be byte-identical across requests — serialize them deterministically, sorted by name), then the frozen core of your system prompt, then any per-session context, then conversation history, and finally the live user query at the very end. The cache checkpoint goes at the boundary between what's shared and what varies.
The most common single mistake is putting dynamic data in the system prompt. "Current date: 2026-07-04," "logged-in user: alice," "mode: expedited" — every one of these interpolated into the system header invalidates the entire prompt on every request, and no amount of cache-control markers will save you, because the bytes genuinely differ each time. The fix is to move that content after the last checkpoint. A date injected at the end of the message array invalidates nothing before it. One team reported that moving their dynamic working memory out of the system prompt and into a trailing user message took their cache hit rate to 84% overnight and cut their bill correspondingly.
For per-tenant or per-user context, the same logic applies at a finer grain: keep the truly global prefix (shared across all tenants) at the top with its own checkpoint, and put the per-tenant fragment below that boundary with a second checkpoint. That way a global prompt update and a tenant-specific value invalidate independently instead of one poisoning the other. Providers give you a budget of breakpoints — Anthropic allows four cache_control markers per request — precisely so you can place them at these stability seams.
There are two subtler traps worth internalizing. Fork operations — a summarizer, a compaction pass, a sub-agent spawned mid-run — must reuse the parent's exact system prompt, tools, and model, or they miss the parent's cache entirely and pay cold. And on long agentic turns, some providers only walk back a limited window (Anthropic looks back at most 20 content blocks) to find a prior cache entry; a turn that appends more than that many tool-call blocks can silently blow past the lookback and miss even though nothing "changed." If your agent loop stuffs 40 tool results into one turn, place an intermediate checkpoint every ~15 blocks.
All of this architecture is worthless if you can't see the cliff coming, and the good news is that the signal is sitting in your API responses already. Every provider reports cache activity in the usage object: on Anthropic it's cache_creation_input_tokens (what you wrote this request) and cache_read_input_tokens (what you read). The ratio of reads to total input is your cache hit rate, and it is the single most sensitive early-warning metric you have.
The FinOps move is to alarm on cache-hit-rate as a first-class SLO, not to reconstruct it from the invoice a month later. A healthy production system with a big shared prefix should show a cache-read share up in the 80–95% range. If that number drops sharply and stays down across repeated identical-prefix requests, a silent invalidator is at work — and it almost always traces to a recent change in something that feeds the prefix. When the alarm fires, the diagnosis is mechanical: diff the rendered prompt bytes between two consecutive requests and look for the first position where they differ. That's your invalidator.
This reframes the whole problem. The prompt-cache cliff isn't really a caching bug; it's a change-management gap. The system prompt is load-bearing cost infrastructure, and right now most teams treat editing it as a trivial content change that anyone can merge without review. It isn't. A one-line append to the top of the prefix has the same fleet-wide blast radius as a config change to your rate limiter — it just doesn't announce itself.
So put the guardrails where the risk is. Gate edits to the cacheable prefix behind a review that knows what caching costs. Wire a cache-hit-rate check into the same dashboard that watches latency and error rate, and alarm on regressions. Structure the prompt so that the stuff people actually need to change lives below the cache boundary, where editing it is cheap and safe by construction. Do that, and the next well-meaning one-line PR lands in the volatile section where it belongs — and your bill doesn't move.
For members
The rest is for members.
Members get the rest — the frameworks, decisions, and reasoning behind every idea I publish in public.
—Complete essays, including the parts I keep off the public archive
—Working frameworks with the trade-offs spelled out
