16 posts tagged with "model-migration"

A new frontier model ships. It's cheaper, faster, and tops every public leaderboard. You run it through your eval suite — the one you've spent eighteen months building — and it scores worse than the model you're running today. So you keep the incumbent, file the challenger under "not ready," and move on.

Here's the uncomfortable part: that result tells you almost nothing about which model is better. It tells you that your eval suite was built by watching your current model fail, one production incident at a time, and then patched to make those specific failures go away. The suite isn't a neutral measurement of quality. It's a catalog of one model's scar tissue. And a challenger that has different weaknesses will always look worse against a test set assembled from the incumbent's particular weaknesses — even when it's better on the traffic you actually serve.

This is incumbent bias, and it's the switching cost nobody prices into the migration decision. It quietly locks you onto a model long after a better option exists, and it does it while wearing the costume of rigorous engineering.

When a dependency breaks, your build fails. When a workaround becomes unnecessary, nothing happens at all. That asymmetry is why every production LLM system older than a year is carrying scaffolding it no longer needs — retry choreography, output-repair parsers, forced chain-of-thought, elaborate task decomposition, chunking heuristics — each one built as a compensation for a specific model's specific weakness, and each one silently outliving the weakness it compensated for.

The uncomfortable part is that this isn't a hygiene problem, like stale feature flags. Obsolete scaffolding doesn't just sit there costing you latency and tokens. In the worst case it actively constrains the new model to the old model's ceiling: your decomposition logic chops a task into six steps because the 2024 model couldn't hold the whole thing, and the 2026 model — which could have one-shotted it — now inherits six opportunities to lose context at the seams you built.

Ask any engineering leader whether they're locked into a model vendor and they'll point you at the abstraction. "We route everything through a gateway. Swapping providers is a config change." The endpoint is one line. The base URL is an environment variable. On paper, migration is a Tuesday afternoon.

Then they try it. They flip the config to a different model family, the integration tests stay green, and production quietly falls apart. The JSON that always parsed now arrives wrapped in a markdown fence. The classifier that hit 94% drops to the low eighties. A prompt that ran clean for a year starts refusing one request in twenty for reasons no one can reproduce. The endpoint swapped in seconds. The behavior did not come with it.

This is the lock-in nobody budgets for. It doesn't live in your contract or your SDK. It lives in your prompts — thousands of small accommodations your team made, one at a time, to the quirks of a single model family. You didn't choose a model. You married one, and the prenup is every prompt you've ever shipped.

The migration looked clean. Evals were re-anchored against the new model version. Judge prompts were re-calibrated. Two weeks of shadow traffic showed behavior parity within tolerance. p50 and p99 latency were inside the budget. The rollout call signed off on Thursday afternoon and the team went home.

By Friday morning, the inference bill was 3x normal. Eval scores were still fine. Latency was still fine. No one on the rollout call had thought to instrument the cache hit rate, because the prefix had not changed — the system prompt was byte-identical, the tool definitions were byte-identical, the conversation framing was byte-identical. What had changed was the model version in the request body, and the provider keys its prefix cache on (prefix bytes + model version). Every request after the cutover landed on a cold cache. The warm-up curve took six weeks of organic traffic to recover, and the team paid full input-token rates for every token on every request for the duration.

A deprecation notice looks harmless. It arrives as a calm paragraph in a changelog or an email: this model snapshot will be removed from the API on a date a few months out, here is the recommended replacement, thank you for building with us. The implied work is a one-line change — swap the model string, redeploy, done.

That framing is wrong, and it is wrong in an expensive way. The model string is the smallest thing you are losing. The thing that actually leaves with the old model is the prompt you spent six months tuning — every edge-case patch, every reordered instruction, every "respond only with valid JSON, do not wrap it in markdown" you added because that specific model did that specific annoying thing. None of that was portable. It was fitted, in the statistical sense, to one model's behavior. The replacement is not bug-for-bug compatible, so the fit no longer holds.

A model end-of-life is a migration project. Treat it as a config change and you will discover the difference in production, on the new model, with real traffic.

Every model upgrade gets sold to the team as a swap: a one-line config change, a measurable win on latency or cost or quality, and a few days of prompt re-tuning to absorb the new model's quirks. The procurement deck shows per-token deltas, the engineering ticket lists the rollout phases, and the FP&A team books the quarterly savings. Then the eval scores come in and nobody recognizes them. Quality is flat where it should have moved. Two judges that used to agree are now diverging by ten points. The snapshot suite is red, but the diffs look like rewordings. Somebody in standup asks the question that should have been on the migration plan from day one: what is the model actually scoring against?

This is the second bill — the eval re-anchoring tax — and it is reliably larger than the first. The human-annotated reference scores were anchored to the previous model's output distribution. The LLM-as-judge graders were calibrated against the old model's failure modes. The snapshot fixtures captured the old model's wording. The team's intuition for "good output" was trained on the old model's stylistic tells. None of that survives the swap intact.

The CTO of an AI startup I talked to last month opened her laptop and showed me a number: 47. That was the count of distinct system prompts running in production, one per enterprise customer or per logical group of them. The base prompt had been forked once in month four for a healthcare customer that needed a softer refusal posture. Then once more for a legal customer that wanted citations. Then for a financial-services customer whose compliance team had a list of forbidden phrases. None of these felt like a big deal at the time. Each was a small ask, approved in isolation, that the account team could close the deal on.

Two years later, the model provider announced the cutover deadline for the version her prompts were tuned against. Her engineering team's first instinct was to run the eval suite against the new model. The eval suite was scoped to the base prompt. The base prompt was still serving customer zero, which had no overrides, and which represented roughly 9% of revenue.

The deprecation email arrives on a Tuesday afternoon. The model your billing pipeline has depended on for fourteen months is now on a sixty-day timer. The prompt was tuned by an engineer who left in March. The eval suite hasn't been re-baselined since launch. The customer-success team is asking why "the AI feels different" on two enterprise accounts. Nobody put this on the roadmap, and nobody will own it cleanly, because in your org's mental model this is a one-off project — even though it is the fourth one this year.

Every team running an AI feature in production runs into the same realization within eighteen months: the foundation-model provider is operating on a deprecation cadence that the team did not plan for, and the team's migration response keeps being a reactive scramble triggered by a notification email. The fix is not a better playbook for the next migration — there are already plenty of those, and your team has probably written one. The fix is to stop treating migration as a project and start treating it as a recurring operational primitive. Put it on the calendar.

When a cost spike, a model deprecation notice, or a competitor's benchmark forces you to swap providers, engineering teams typically evaluate the candidate on capability benchmarks and call it a migration plan. That process catches about half the problems. The other half aren't capability problems — they're behavioral ones: the invisible layer of formatting habits, refusal patterns, serialization quirks, and output conventions your production code has silently wired itself to over months of iteration.

The capability benchmark tells you whether the new model can do the task. The behavioral fingerprint tells you whether your codebase can survive the replacement.

When OpenAI tried to pull GPT-4o from ChatGPT in August 2025, the backlash was strong enough — organized hashtags, paying users threatening to cancel, public reversal within days — that the company restored it as a default option and promised "substantial notice" before any future removals. The replacement was, by every benchmark the team cared about, better. None of that mattered. Users had spent months learning the model's quirks, calibrating their judgment to its failure modes, and integrating its specific phrasing into workflows the team had never instrumented. Replacing it with "the better version" reset that calibration to zero.

This is the failure mode that the standard deprecation playbook does not cover. Sunsetting a regular SaaS feature — announce, migrate, dark-launch the removal, retire — assumes the user contract is the API surface. For AI features, the contract is the observed behavior of the model: phrasings, tendencies, failure modes, the specific way it handles ambiguity. Users build scaffolding on top of that behavior, and most of the scaffolding lives in their heads, on their laptops, and in downstream systems your team never touches.

The team that treats "we use the latest model" as a virtue is one sunset email away from a quarter of unplanned work. By the time the deprecation notice lands, the architectural decisions that determine whether you can absorb it have already been made — months ago, by people who weren't thinking about migrations at all. The eval suite was implicitly trained against a specific checkpoint. The prompts were tuned against a specific refusal style. The cost projections assumed a specific token-per-task baseline. The router has a hardcoded fallback to a model that is itself about to disappear. None of these decisions look like risks until the email arrives, and then all of them look like the same risk.

Model deprecation is now the most predictable surprise in the AI stack. Anthropic gives a minimum of 60 days' notice on publicly released models. OpenAI's notice windows range from three months for specialized snapshots to 18 months for foundational models, but in practice a recent batch of ChatGPT model retirements landed with as little as two weeks' warning for some teams. GitHub deprecated a slate of Anthropic and OpenAI models in February 2026 in a single coordinated changelog entry. The pattern is no longer "if a model retires" — it's "every quarter, at least one model your stack depends on enters a retirement window, and the calendar isn't synchronized to your roadmap."

Pinning a model version in production feels like engineering discipline. You lock claude-opus-4-0 or gpt-4o-2024-08-06 into config, write a note in the README about why, and move on to shipping features. The output distribution stops shifting under you, the evals stay green, and the prompt tuning you did last quarter keeps working. What you've actually done is start a silent timer. Twelve to fifteen months later the deprecation email arrives, and three sprints of undocumented behavioral dependencies — prompt tuning, eval calibration, output shape assumptions, temperature quirks — all come due at once.

This is the ship-and-pin trap. Pinning is correct in the short term and catastrophic in the long term, because the cost of stability compounds in places you aren't looking. The prompt that was "good enough" a year ago is now load-bearing in ways nobody documented. The JSON schema your downstream service expects was shaped to one model's tokenization habits. The few-shot examples you hand-tuned were tuned against a specific model's notion of helpfulness. When the provider retires the version string, none of these dependencies migrate automatically, and the work to re-qualify them always lands under deadline pressure.