The Model Migration Playbook: How to Swap Foundation Models Without a Feature Freeze

Every production LLM system will face a model migration. The provider releases a new version. Your costs need to drop. A competitor offers better latency. Regulatory requirements demand a different vendor. The question is never if you'll swap models — it's whether you'll do it safely or learn the hard way that "just run the eval suite" leaves a crater-sized gap between staging confidence and production reality.

Most teams treat model migration like a library upgrade: swap the dependency, run the tests, ship it. This works for deterministic software. It fails catastrophically for probabilistic systems where the same input can produce semantically different outputs across model versions, and where your prompt was implicitly tuned to the behavioral quirks of the model you're replacing.

Why Model Migrations Are Different From Software Upgrades

When you upgrade a database driver, the contract is explicit: same query, same results. LLM migrations break this assumption in three ways that compound on each other.

Prompt-model co-adaptation. Your prompts evolved alongside your current model. Every time an engineer tweaked a system prompt to fix an edge case, they were implicitly encoding that model's interpretation style into the prompt. A prompt that reliably produces structured JSON from GPT-4 may produce markdown-wrapped JSON from Claude, or add unsolicited commentary from Gemini. These aren't bugs in the new model — they're the residue of optimization for the old one.

Behavioral surface area. A model isn't just its accuracy on your eval set. It's how it handles ambiguity, how aggressively it refuses edge cases, how it formats outputs when the schema isn't perfectly specified, how it responds to adversarial inputs. Two models that score identically on a benchmark can behave completely differently on the 15% of production traffic that doesn't look like your eval distribution.

Downstream coupling. Your post-processing code, your guardrails, your output parsers — they all learned to expect specific patterns from your current model. A model that wraps JSON in triple backticks, or that spells out numbers instead of using digits, or that adds a preamble before the structured response, breaks downstream systems in ways that unit tests never anticipated.

The Shadow Period: Dual-Write Before You Cut Over

The single most important technique in model migration is the shadow deployment period. Run the new model on live production traffic without serving its responses to users. Log everything. Compare everything. Only cut over when the delta is understood and acceptable.

Here's how to structure it:

Phase 1: Parallel execution (1-2 weeks). Route a copy of every production request to the new model. Store both responses with a correlation ID. Users only see responses from the existing model. This costs you extra inference spend, but it's the cheapest insurance you'll buy.

Phase 2: Automated comparison. Build comparison pipelines that go beyond string matching. You need semantic similarity scores between old and new outputs, structural conformance checks (does the new model's JSON parse correctly?), and behavioral classification (did the new model refuse a request the old model handled, or vice versa?).

Phase 3: Discrepancy triage. Not all differences are regressions. The new model might produce better answers on some inputs. The goal isn't zero difference — it's zero surprising difference. Classify discrepancies into:

• Improvements: new model is demonstrably better
• Neutral variants: semantically equivalent, differently worded
• Regressions: new model fails where old model succeeded
• Novel behaviors: new model does something neither expected nor covered by evals

The novel behaviors category is the one that bites hardest. These are the cases your eval suite never anticipated because the old model never triggered them.

Prompt Translation Is Engineering, Not Find-and-Replace

When teams migrate between model families — say, from OpenAI to Anthropic, or from a proprietary model to an open-source one — the prompt layer requires genuine re-engineering. Each model family has different preferences for how instructions are structured, how context is delimited, and how output format is specified.

OpenAI models respond well to markdown-formatted prompts with section delimiters. Claude prefers XML tags for separating instructions from context. Gemini emphasizes explicit constraint placement. These aren't cosmetic differences — they affect how reliably the model follows complex multi-step instructions.

The practical migration pattern:

1. Extract the intent from each prompt. What is this prompt actually trying to achieve? Strip away the model-specific formatting and write down the core requirements.
2. Re-express for the target model. Use the target model's preferred structuring conventions. This often means rewriting, not translating.
3. Test on your hardest cases first. Don't start with the happy path. Start with the prompts that required the most iteration to get right on the old model — those are the ones most likely to break.
4. Preserve the escape hatches. If your old prompt had specific instructions for handling edge cases ("if the user asks about X, respond with Y"), verify these still trigger correctly. Edge case handling is where model-specific tuning concentrates.

A common mistake is building an abstraction layer that "normalizes" prompts across providers. This sounds elegant but produces mediocre results everywhere. The best prompts are model-specific. Accept the maintenance cost of per-model prompt variants in exchange for per-model reliability.

Embedding Migration: The Reindexing Problem Nobody Budgets For

If your system uses embeddings — for RAG, semantic search, or classification — a model migration creates a second, often larger migration: your entire vector index becomes incompatible. New embedding models produce vectors in a different semantic space. You cannot mix old and new embeddings in the same index and expect meaningful similarity scores.

Loading…