The Tokenizer Upgrade That Invalidated Every Prompt Cache Prefix

The release notes were two lines long. "Improved multilingual tokenization. No breaking changes to model outputs." Nine words. Your evals confirmed it: same prompts, same completions, same scores. Your platform team signed off on the upgrade Friday afternoon. By Tuesday morning your cache hit rate had collapsed from 80% to 4%, your daily inference bill had quadrupled, and the on-call engineer who paged you at 6am could not find a single line of your code that had changed.

Nothing in your code had changed. The provider had shipped a new tokenizer that split one Unicode glyph one byte differently than the old one. Every cached prefix in your system was now fingerprinted against a token sequence that no longer existed. The model behaved identically — that was true. The cache layer, which the release notes did not mention, paid the bill in full.

This is the failure mode that prompt caching makes structural: the contract you depend on for cost is not the contract the provider tests against. The provider tests that the model produces the same output for the same input. You depend on the model producing the same tokens for the same input, in the same order, with the same byte boundaries, hashed into the same prefix fingerprint. Those are different contracts, and only one of them has a regression suite.

What the cache actually keys on

Prompt caching, as deployed by every major provider, is a prefix cache. The system hashes the token sequence from the very first token forward, in chunks, and stores intermediate KV state keyed by those hashes. When a new request comes in, the cache walks forward from the start, hashing tokens until it finds a chunk boundary that does not match a stored hash. Everything up to that point is a hit. Everything after it is recomputed at full price.

The keying detail that matters: the cache hashes tokens, not characters. The provider tokenizes your raw input before computing the hash. If the tokenizer changes — if the same string now produces a different token sequence — every cached prefix becomes unreachable. The strings on disk are identical. The fingerprints derived from them are not.

For most prompt content this is not a problem, because tokenizers are stable for the vast majority of ASCII text and well-established multilingual content. The hazard lives at the edges:

• Unicode characters that can be represented multiple ways under different normalization forms (NFC vs NFD), where é is either one codepoint or two
• Combining marks, emoji sequences with zero-width joiners, and regional indicators
• Glyphs from less common scripts where the tokenizer's BPE merges were retrained against a different corpus
• Byte sequences that straddle pre-tokenization boundaries the new tokenizer draws differently

A tokenizer "upgrade" — usually framed as better multilingual coverage or a larger vocabulary — almost always changes some of these edges. The model behavior is preserved by design: training continues from the new tokenizer with enough data that the output distribution matches. The cache fingerprint is preserved by accident, and only for content that happens to land on identical token boundaries before and after the upgrade. For anything else, the prefix breaks.

Why this is invisible until the bill arrives

The reason this fails silently is that the failure mode is correctness-preserving. Every request still works. Every completion is still right. The eval suite passes. The latency budget might widen a little because cache hits return faster than cache misses, but cache hits also have variance, so a 20–40% latency increase on the long tail looks like normal noise unless someone is watching the distribution. The only thing that breaks is the cost line, and the cost line is reported on a 24-hour delay against a budget that has slack baked in for traffic growth.

The standard telemetry stack does not catch this. Most teams instrument three things on their LLM calls: latency, error rate, and total token count. None of those move during a tokenizer-driven cache collapse. The cached prefix that used to bill at 10% of full price now bills at 100%, and the request paths exposing that line item — cached_input_tokens versus input_tokens — are usually aggregated into a single "input tokens" metric on the dashboard. You can have a complete cache failure and never see it on the graph that the on-call person is watching.

Loading…