13 posts tagged with "model-migration"

The migration looked clean. Evals were re-anchored against the new model version. Judge prompts were re-calibrated. Two weeks of shadow traffic showed behavior parity within tolerance. p50 and p99 latency were inside the budget. The rollout call signed off on Thursday afternoon and the team went home.

By Friday morning, the inference bill was 3x normal. Eval scores were still fine. Latency was still fine. No one on the rollout call had thought to instrument the cache hit rate, because the prefix had not changed — the system prompt was byte-identical, the tool definitions were byte-identical, the conversation framing was byte-identical. What had changed was the model version in the request body, and the provider keys its prefix cache on (prefix bytes + model version). Every request after the cutover landed on a cold cache. The warm-up curve took six weeks of organic traffic to recover, and the team paid full input-token rates for every token on every request for the duration.

A deprecation notice looks harmless. It arrives as a calm paragraph in a changelog or an email: this model snapshot will be removed from the API on a date a few months out, here is the recommended replacement, thank you for building with us. The implied work is a one-line change — swap the model string, redeploy, done.

That framing is wrong, and it is wrong in an expensive way. The model string is the smallest thing you are losing. The thing that actually leaves with the old model is the prompt you spent six months tuning — every edge-case patch, every reordered instruction, every "respond only with valid JSON, do not wrap it in markdown" you added because that specific model did that specific annoying thing. None of that was portable. It was fitted, in the statistical sense, to one model's behavior. The replacement is not bug-for-bug compatible, so the fit no longer holds.

A model end-of-life is a migration project. Treat it as a config change and you will discover the difference in production, on the new model, with real traffic.

Every model upgrade gets sold to the team as a swap: a one-line config change, a measurable win on latency or cost or quality, and a few days of prompt re-tuning to absorb the new model's quirks. The procurement deck shows per-token deltas, the engineering ticket lists the rollout phases, and the FP&A team books the quarterly savings. Then the eval scores come in and nobody recognizes them. Quality is flat where it should have moved. Two judges that used to agree are now diverging by ten points. The snapshot suite is red, but the diffs look like rewordings. Somebody in standup asks the question that should have been on the migration plan from day one: what is the model actually scoring against?

This is the second bill — the eval re-anchoring tax — and it is reliably larger than the first. The human-annotated reference scores were anchored to the previous model's output distribution. The LLM-as-judge graders were calibrated against the old model's failure modes. The snapshot fixtures captured the old model's wording. The team's intuition for "good output" was trained on the old model's stylistic tells. None of that survives the swap intact.

The CTO of an AI startup I talked to last month opened her laptop and showed me a number: 47. That was the count of distinct system prompts running in production, one per enterprise customer or per logical group of them. The base prompt had been forked once in month four for a healthcare customer that needed a softer refusal posture. Then once more for a legal customer that wanted citations. Then for a financial-services customer whose compliance team had a list of forbidden phrases. None of these felt like a big deal at the time. Each was a small ask, approved in isolation, that the account team could close the deal on.

Two years later, the model provider announced the cutover deadline for the version her prompts were tuned against. Her engineering team's first instinct was to run the eval suite against the new model. The eval suite was scoped to the base prompt. The base prompt was still serving customer zero, which had no overrides, and which represented roughly 9% of revenue.

The deprecation email arrives on a Tuesday afternoon. The model your billing pipeline has depended on for fourteen months is now on a sixty-day timer. The prompt was tuned by an engineer who left in March. The eval suite hasn't been re-baselined since launch. The customer-success team is asking why "the AI feels different" on two enterprise accounts. Nobody put this on the roadmap, and nobody will own it cleanly, because in your org's mental model this is a one-off project — even though it is the fourth one this year.

Every team running an AI feature in production runs into the same realization within eighteen months: the foundation-model provider is operating on a deprecation cadence that the team did not plan for, and the team's migration response keeps being a reactive scramble triggered by a notification email. The fix is not a better playbook for the next migration — there are already plenty of those, and your team has probably written one. The fix is to stop treating migration as a project and start treating it as a recurring operational primitive. Put it on the calendar.

When a cost spike, a model deprecation notice, or a competitor's benchmark forces you to swap providers, engineering teams typically evaluate the candidate on capability benchmarks and call it a migration plan. That process catches about half the problems. The other half aren't capability problems — they're behavioral ones: the invisible layer of formatting habits, refusal patterns, serialization quirks, and output conventions your production code has silently wired itself to over months of iteration.

The capability benchmark tells you whether the new model can do the task. The behavioral fingerprint tells you whether your codebase can survive the replacement.

When OpenAI tried to pull GPT-4o from ChatGPT in August 2025, the backlash was strong enough — organized hashtags, paying users threatening to cancel, public reversal within days — that the company restored it as a default option and promised "substantial notice" before any future removals. The replacement was, by every benchmark the team cared about, better. None of that mattered. Users had spent months learning the model's quirks, calibrating their judgment to its failure modes, and integrating its specific phrasing into workflows the team had never instrumented. Replacing it with "the better version" reset that calibration to zero.

This is the failure mode that the standard deprecation playbook does not cover. Sunsetting a regular SaaS feature — announce, migrate, dark-launch the removal, retire — assumes the user contract is the API surface. For AI features, the contract is the observed behavior of the model: phrasings, tendencies, failure modes, the specific way it handles ambiguity. Users build scaffolding on top of that behavior, and most of the scaffolding lives in their heads, on their laptops, and in downstream systems your team never touches.

The team that treats "we use the latest model" as a virtue is one sunset email away from a quarter of unplanned work. By the time the deprecation notice lands, the architectural decisions that determine whether you can absorb it have already been made — months ago, by people who weren't thinking about migrations at all. The eval suite was implicitly trained against a specific checkpoint. The prompts were tuned against a specific refusal style. The cost projections assumed a specific token-per-task baseline. The router has a hardcoded fallback to a model that is itself about to disappear. None of these decisions look like risks until the email arrives, and then all of them look like the same risk.

Model deprecation is now the most predictable surprise in the AI stack. Anthropic gives a minimum of 60 days' notice on publicly released models. OpenAI's notice windows range from three months for specialized snapshots to 18 months for foundational models, but in practice a recent batch of ChatGPT model retirements landed with as little as two weeks' warning for some teams. GitHub deprecated a slate of Anthropic and OpenAI models in February 2026 in a single coordinated changelog entry. The pattern is no longer "if a model retires" — it's "every quarter, at least one model your stack depends on enters a retirement window, and the calendar isn't synchronized to your roadmap."

Pinning a model version in production feels like engineering discipline. You lock claude-opus-4-0 or gpt-4o-2024-08-06 into config, write a note in the README about why, and move on to shipping features. The output distribution stops shifting under you, the evals stay green, and the prompt tuning you did last quarter keeps working. What you've actually done is start a silent timer. Twelve to fifteen months later the deprecation email arrives, and three sprints of undocumented behavioral dependencies — prompt tuning, eval calibration, output shape assumptions, temperature quirks — all come due at once.

This is the ship-and-pin trap. Pinning is correct in the short term and catastrophic in the long term, because the cost of stability compounds in places you aren't looking. The prompt that was "good enough" a year ago is now load-bearing in ways nobody documented. The JSON schema your downstream service expects was shaped to one model's tokenization habits. The few-shot examples you hand-tuned were tuned against a specific model's notion of helpfulness. When the provider retires the version string, none of these dependencies migrate automatically, and the work to re-qualify them always lands under deadline pressure.

A healthcare company running a production AI triage assistant gets the email every team dreads: their inference provider is retiring the model they're using in 90 days. They update the model string, run a quick manual smoke test, and ship the replacement. Three weeks later, the new model starts offering unsolicited diagnostic opinions. Token usage explodes 5×. Entire prompt templates break because the new model interprets instruction phrasing differently. JSON parsing fails because the output schema shifted.

This is not an edge case. It is the normal experience of surviving a model retirement when you treat it as a configuration change rather than a systems migration.

Most prompt migrations look fine in staging. Ninety percent of test cases pass, the new model's responses feel crisper, and the demo runs cleanly. Then you ship, and within two days your structured output parser is throwing exceptions on 12% of responses, a customer-facing classification pipeline started returning wrong labels, and a tool-calling agent is looping on a schema it used to handle without issue. Nobody changed the prompts. The model changed.

This is the prompt-model coupling trap: prompts that work reliably on one model silently accumulate dependencies on that model's specific behavioral quirks, and those dependencies are invisible until migration day.

Most teams discover they are model-dependent the same way you discover a load-bearing wall — by trying to remove it. The deprecation email arrives, you swap the model identifier in your config, and your application starts returning confident, well-formatted, subtly wrong answers. No errors. No crashes. Just a slow bleed of trust that takes weeks to notice and months to repair.

This is the model deprecation cliff: the moment when a forced migration reveals that your "model-agnostic" system was never agnostic at all. Your prompts, your output parsers, your evaluation baselines, your users' expectations — all of them were quietly calibrated to behavioral quirks that are about to change on someone else's release schedule.