The Hidden Switching Costs of LLM Vendor Lock-In

Most engineering teams believe they've insulated themselves from LLM vendor lock-in. They use LiteLLM to unify API calls. They avoid fine-tuning on hosted platforms. They keep raw data in their own storage. They feel safe. Then a provider announces a deprecation — or a competitor's pricing drops 40% — and the team discovers that the abstraction layer they built handles roughly 20% of the actual switching cost.

The other 80% is buried in places no one looked: system prompts written around a model's formatting quirks, eval suites calibrated to one model's refusal thresholds, embedding indexes that become incompatible the moment you change models, and user expectations shaped by behavioral patterns that simply don't transfer.

This is a map of that 80%.

API Compatibility Is the Easy Part

When engineers think about vendor lock-in, they usually think about APIs. OpenAI uses /chat/completions. Anthropic uses /messages. The parameters are slightly different. Tools/functions have different schemas. This is the well-understood surface.

Unified gateway tools like LiteLLM, Portkey, and provider-neutral SDKs like LangChain solve this layer competently. You swap the model parameter, maybe adjust the message format, and your code runs.

What they do not solve is behavioral compatibility — and that's where the cost lives.

When you switch from GPT-4o to Claude, or from Anthropic to Gemini, the API surface changes in an afternoon. Everything downstream takes weeks.

The Prompt Portability Problem

Every frontier model has behavioral fingerprints that prompt engineers learn to exploit. Claude responds exceptionally well to XML-tagged structure — <instructions>, <document>, <examples> — as a way to separate concerns. GPT models respond better to Markdown with headers. Gemini tends toward structured reasoning if you ask for it explicitly.

These aren't just stylistic differences. They compound in complex prompts:

• A 2,000-token system prompt tuned for Claude's XML parsing will produce degraded results when dropped into GPT-4o, not because GPT-4o can't follow instructions, but because the structural cues are mismatched.
• Prompts that rely on Claude's tendency toward concision produce unexpectedly verbose outputs on GPT models.
• JSON extraction prompts that work reliably on GPT-4o (which has a natural bias toward JSON) require explicit instruction reinforcement on models without that bias.

None of this surfaces in a quick API test. It shows up in your p5 failure cases: the edge inputs, the long documents, the ambiguous queries. A model swap that looks clean in a demo will hemorrhage accuracy in production at the tail.

Sustainable prompt patterns — clear goal statements, explicit output specifications, few-shot examples demonstrating desired behavior — do port across models. But most production prompts aren't built that way. They're built iteratively, incorporating model-specific tweaks accumulated over months. Disentangling the portable logic from the model-specific tuning is not a refactor — it's a rewrite.

Your Evals Are Measuring the Wrong Model

Custom evaluation suites are the second invisible lock-in layer, and arguably the most dangerous because teams trust them.

A well-maintained eval suite feels like the objective arbiter of model quality. When Provider B outperforms Provider A on public benchmarks, you run your evals and see Provider A win. This isn't a contradiction — it's a signal that your evals are testing model-specific behavior rather than business requirements.

Common patterns that create eval lock-in:

Format-based assertions. An eval checking that "JSON is returned 95% of the time" is measuring GPT-4o's default output tendency, not your actual requirement. When you switch to a model with different formatting defaults, the eval fails — but your product might work fine if you add a format instruction.

Calibrated thresholds. Teams set accuracy thresholds like "> 87% on the classification task" by observing what their current model achieves. These thresholds encode one model's performance envelope, not an independent standard. A better model might score 91% — or 84% with different failure modes that your threshold doesn't distinguish.

Refusal-sensitive test cases. If your eval includes edge cases near the boundaries of your current model's refusal behavior, switching providers changes pass/fail on those cases immediately. Whether this is a regression depends entirely on which side of the refusal boundary your use case actually needs.

The fix isn't more evals. It's eval design that starts from user outcomes and business requirements, not from observing what the current model does and treating that as the specification.

Embedding Lock-In: The Rebuild Nobody Budgets For

Of all the switching cost categories, embedding model lock-in has the most predictable structure and the most consistently underestimated cost.

Every embedding model creates its own vector space. A 1,536-dimensional vector from OpenAI's text-embedding-3-large has no meaningful geometric relationship to a 1,536-dimensional vector from Cohere's embed-english-v3.0, even if both encode the same sentence. The spaces are topologically unrelated.

Switching embedding providers requires:

1. Full corpus re-embedding. Every document in your retrieval index must be processed through the new model. For a system with 50 million documents, this is tens of thousands of dollars in compute and several days of pipeline time.

2. Vector index rebuild. The HNSW graphs, IVF partitions, or other index structures in your vector database are optimized for the old embedding distribution. They must be rebuilt from scratch on the new vectors.

3. Zero-downtime orchestration. You can't swap in-place. You need parallel index operation — serving queries against the old index while building the new one, then cutting over. This is non-trivial operational work.

4. Retrieval quality re-validation. Your retrieval benchmarks were developed against the old model's semantic groupings. After the switch, you need to revalidate that your search quality is maintained or improved.

Teams that use vector database collection aliases — where the application references a semantic name rather than a specific collection identifier — can cut over in seconds once rebuilding is complete. Teams that hardcode collection identifiers face additional migration work on top of the rebuild cost.

The practical implication: treat your embedding model choice with the same weight as your primary database choice. It's not infrastructure you swap casually.

Refusal Boundaries Shape User Behavior

A less-discussed but product-critical form of lock-in is refusal policy. Frontier models differ substantially in where they draw the line:

Anthropic's Claude has predictable, consistent refusal behavior trained through Constitutional AI. OpenAI's models are more likely to offer alternatives rather than flat refusals on borderline requests. Google's Gemini models have shown significant variation between versions — one analysis found fulfillment rates on certain content categories dropping from 33% to 7% between major versions.

For content-adjacent applications — writing tools, customer service bots, research assistants — these differences are invisible in a demo and visible in production. A customer service agent that handles 10,000 queries per day will refuse hundreds of legitimate queries per day if its refusal threshold shifts even slightly after a model swap.

Users who experience more refusals don't file support tickets explaining that refusal rates increased. They abandon tasks. Session completion rates drop. Support volume increases. The root cause shows up in usage analytics weeks later, by which point causality is hard to establish.

The Tokenizer Tax

A subtle cost that compounds across the other categories: tokenizers differ between providers, and those differences affect cost estimates, prompt length calculations, and context window planning.

The same input text produces different token counts on OpenAI's tiktoken vs. Anthropic's tokenizer. Anthropic's tokenizer tends to produce more tokens from the same text. This means:

• Your context window planning is off. A prompt designed to fit within 4,096 tokens on GPT-4o may exceed that limit on Claude's tokenizer.
• Your cost estimates are wrong. If you switch providers expecting equivalent cost at equivalent token counts, the tokenizer difference will silently inflate your bills.
• Rate limits behave differently. Token-based rate limits (tokens per minute) interact with tokenizer efficiency, so your peak load behavior changes even if request patterns don't.

This isn't a migration-stopper, but it's the kind of thing that surfaces as mysterious production behavior — the prompt that works in testing starts getting truncated inconsistently, or costs are 15% higher than projected — and takes time to diagnose if you don't know to look for it.

What Actually Survives a Model Swap

Not everything requires a rewrite. Some patterns are genuinely portable:

Clear, goal-oriented instructions. Prompts structured around explicit task goals ("You are reviewing the following code change for security vulnerabilities. Identify...") transfer better than prompts structured around working around a model's weaknesses ("Never add preambles, just respond directly with...").

Few-shot examples. Showing desired behavior through examples is more portable than explaining it through instructions. Most frontier models extract the pattern from examples efficiently.

Explicit output schemas. Specifying the exact structure of the expected output — with types, required fields, and examples — survives provider switches better than relying on a model's default output tendencies.

Role framing. Persona instructions ("You are a senior engineer reviewing a PR") are sufficiently abstract that they port cleanly.

Business logic in data, not in prompts. The most portable system design moves business rules into structured context — lookup tables, configuration, retrieved documents — rather than encoding them in prompt language that a model might interpret differently.

Mitigation That Actually Works

The abstraction-layer approach (LiteLLM, provider-neutral SDKs) addresses the API surface, which is real but shallow. The deeper mitigations require architectural decisions made earlier:

Own your data. Training data, eval sets, fine-tuning datasets, and prompt libraries should live in version-controlled repositories you control. When you fine-tune on a hosted platform and the weights live in their infrastructure, you've lost the ability to reproduce that capability elsewhere.

Design evals for your task, not your model. Write eval cases that express what correct behavior looks like from a user perspective, without reference to what your current model does. This is harder and slower, but it's the only kind of eval that gives you honest signal when switching providers.

Version control your prompts. Treat prompts as first-class software artifacts with version history, change review, and regression testing. This discipline pays dividends during any model transition — you can trace when behavior changed and why.

Plan embedding migration costs upfront. Before selecting an embedding model, build the migration path: collection aliases in your vector database, re-indexing pipelines, retrieval quality benchmarks that run independently of which model produced the embeddings.

Maintain a compatibility test harness. Keep a lightweight suite of test cases that runs against any provider and validates fundamental behavior: JSON format compliance, refusal behavior on edge cases, output length distribution. This doesn't replace your eval suite — it gives you an early warning system when a switch introduces unexpected behavioral changes.

The Deprecation Timeline Is Not Optional

One reality teams underestimate: you will switch providers eventually, whether you choose to or not.

OpenAI retired GPT-4o from the API in 2026 with roughly 12 months of notice. Anthropic deprecated Claude 3.5 Sonnet in August 2025. Most frontier models have effective production lifespans of 12–18 months before deprecation.

This means the question isn't "should we maintain provider portability?" It's "do we absorb migration costs on our schedule, or on our provider's?" Teams that treat their first migration as an opportunity to build portable infrastructure pay the one-time architectural cost and handle future migrations in days. Teams that migrate reactively pay the full switching cost on every deprecation cycle.

The engineering effort is the same either way. The difference is whether you're in control of when it happens.

The Actual Cost

The cost to switch LLM providers for a production system that's been running for a year is roughly:

• 40–70% of the effort in data preparation and eval rebuild
• 20–30% in prompt porting and validation
• 10–20% in infrastructure reconfiguration and operational work

The last category is the one LiteLLM handles. The first two are yours regardless of what abstractions you use.

For a system managed by a senior engineer full-time, that's 4–6 weeks of focused work, assuming the codebase is well-organized and evals exist. For systems with accumulated prompt debt, undocumented behavior, or missing evals — which describes most production AI systems — it's longer.

Provider-agnostic abstractions are worth using. They handle real complexity. But treat them as a toolbelt, not a guarantee. The portability that matters lives in how you design your prompts, structure your evals, and manage your embedding infrastructure — none of which any library can do for you.