Two Model Vendors, One Feature: The Redundancy That Buys a Consistency Nightmare

You wired up a second model vendor because the first one went down. It was a Tuesday, the primary provider's status page was a wall of red for a few hours, and your feature was dead in the water with it. So you did the responsible thing: you added a fallback. Now, if OpenAI is unreachable, you route to Anthropic. If Anthropic rate-limits you, you fall back to Gemini. The architecture diagram looks clean and grown-up. Reliability, solved.

Except you didn't add a replica. You added a second opinion. And a second opinion is a very different thing to operate than a second copy.

The mental model you imported comes from the stateless-service playbook: run three identical instances behind a load balancer, and if one dies the other two serve the exact same responses. That works because the replicas are byte-for-byte interchangeable. Two language models from two vendors are not. They were trained on different data, tuned with different objectives, and they disagree — systematically, not randomly — on exactly the inputs where your users notice.

The failure you were protecting against was a provider outage. The failure you just introduced is subtler and permanent: the same user, asking the same question, gets a materially different answer depending on which "brain" happened to serve the request. You traded a rare, loud problem for a constant, quiet one.

Two Models Are Not Two Replicas

The most seductive part of the microservice analogy is the word "failover." In a stateless system, failover is free and invisible: the second replica does not know or care that the first one died, and neither does your user. The pitch for multi-vendor LLM setups borrows that vocabulary wholesale, and it is misleading.

Start with the fact that even models with identical benchmark scores answer individual questions differently. Recent analysis of cross-model disagreement found that two models with nearly the same overall accuracy can still diverge substantially on specific items, and that this divergence is systematic rather than noise — it reflects genuine differences in how each model represents and reasons about a problem, even under fixed prompts and deterministic decoding. Aggregate accuracy tells you the models are "equally good." It tells you nothing about whether they agree on the request in front of you right now. For a summarizer or a classifier that runs behind a feature, per-item agreement is the thing that actually matters, and it is precisely the thing the benchmark hides.

Then there is format. One engineer ran the same fifty production prompts through six models and logged what varied. The headline numbers weren't about correctness — they were about shape. Asked to "return only JSON, no markdown," one model wrapped its output in code fences roughly a third of the time; another did it closer to 40% of the time, even with the explicit instruction. One model added schema fields nobody requested. Another returned an array where a single object was expected. When the token limit was hit, some models stopped cleanly while others emitted truncated JSON with missing closing braces. If your parser was written against the primary model's clean output, the fallback model's output will break it — not always, just often enough to page someone at 2 a.m.

This is the core of the problem. Your failover path is the least-tested code path in your system, and it only executes when your primary is already on fire. That is the worst possible moment to discover that your backup model formats dates differently, refuses borderline requests the primary would have answered, or silently follows a contradictory instruction the primary would have flagged.

The Prompt You Tuned Is Not Portable

Here is the assumption that quietly wrecks multi-vendor setups: that a prompt is a portable artifact you write once and point at any model. It is not. A prompt is a program compiled against a specific runtime, and every vendor's runtime has different calling conventions.

Anthropic's models are trained to pay close attention to XML-style tags as structural delimiters; a prompt that leans on them gets crisp behavior from Claude and indifferent behavior from a model that treats them as decoration. OpenAI's models respond to a different set of cues. The techniques that make a prompt sing on one platform range from irrelevant to counterproductive on another, and the research backs up the folk wisdom: even the strongest frontier models are demonstrably not robust to prompt phrasing, and that sensitivity is model-specific. Studies of prompt stability have measured double-digit swings in output quality from nothing more than reordering the same information or rephrasing a semantically identical instruction.

The practical consequence is that there is no such thing as "the prompt for this feature" once you have two vendors. There are two prompts — sometimes two prompt families — that happen to aim at the same behavior. The industry even has a name for the work of porting a prompt from one model to another: LLM migration, "the process of re-engineering prompts for different LLMs to achieve the same task." It is engineering work, not a config change.

Which means every time a product manager asks for a tweak — "make the tone warmer," "always include a caveat when the answer is uncertain" — you don't make one change. You make it twice, in two dialects, and then you verify it twice. Which brings us to the bill.

The Dual-Eval Tax Nobody Budgeted For

The cost of a backup model is not the backup model. It's the discipline required to keep it correct.

If you take evaluation seriously — and if you're running a feature important enough to need failover, you should — then every prompt change has to clear your eval suite before it ships. With one vendor, that's one suite, one axis. With two, every change has to pass twice, against two sets of model-specific quirks, or you're flying blind on the exact path you built for emergencies. One engineering team that runs this discipline reported that evaluating prompts across providers added on the order of 20–30% to their development time, because every prompt change had to be evaluated against at least the first two providers before it could ship.

That number is the honest price of the "warm backup." It is not a one-time integration cost; it is a permanent tax on every future change to the feature. And it compounds with prompt drift: a series of small, reasonable fixes tuned to make the primary model behave can silently degrade the secondary, because you were only looking at the primary when you made them. Each change seems innocuous. The cumulative divergence is not.

The cost shows up in your actual bill, too, in a way the resilience story never mentions. "Equivalent" model tiers are not equivalently priced — output tokens in particular can differ by a wide margin across vendors, and in extreme cases the spread between a budget model and a premium one for the same workload runs to an order of magnitude or more. That means your per-request cost is now nondeterministic: the same query costs a different amount depending on which vendor served it. Your finance team's forecast just inherited the same variance as your outputs.

The Streaming Wall: Where Failover Quietly Fails

There's one more detail that punctures the "seamless failover" story, and it's worth internalizing because it's structural, not a bug you can fix.

Failover only works cleanly before the first token. Once you have started streaming tokens to a user's screen — which, for any chat-style feature, is almost immediately — you cannot transparently retry on a different model. As one team building exactly this put it: once they start sending tokens to a user, they can't easily retry with a different model. Restarting mid-stream means the user watches half an answer evaporate and a different one grow in its place. So the real behavior of most "automatic failover" is narrower than the diagram implies: it catches connection errors and pre-generation failures, and it is helpless against a model that starts strong and degrades, or a provider that accepts your request and then stalls halfway through generation.

Your resilience mechanism, in other words, has a consistency hole built into its most common execution path. That's not an argument against having it. It's an argument for knowing exactly what it does and does not buy you, so you stop treating it as a magic uptime multiplier.

Build the Seam, or Don't Bother

None of this means multi-vendor redundancy is wrong. It means it is a real engineering commitment with a real abstraction cost, and you should pay that cost deliberately or not take it on at all.

The failure mode to avoid is the one where provider-specific logic leaks into your domain code — if (provider === 'openai') { ... } else if (provider === 'gemini') { ... } metastasizing through your business logic until every feature knows about every vendor's quirks. The discipline that works is the classic adapter seam: define your own internal contract for what a model call looks like — request shape, response shape, token accounting — and force every vendor's SDK to translate into it behind an adapter. Your feature code talks only to your contract. Adding or swapping a provider becomes a config change at a registry, not a surgery across the codebase. Treat external systems as external; your platform should only ever work with its own internal contract.

But be honest about the one thing the seam cannot hide: structured output. "Give me JSON matching this schema" is not one feature across vendors — it is several different mechanisms wearing the same name. Some providers offer true server-side strict-schema enforcement. Anthropic's models have no loose JSON mode at all and route structured output through tool-calling, so you define a tool, force its use, and extract arguments from the tool-use block. Some providers guarantee only that the output is syntactically valid JSON and leave schema validation to you on the client. A clean adapter can paper over the calling conventions, but somebody still has to write and test the per-provider path that turns "I want this shape" into reality, and that path is where "the same feature" quietly behaves differently.

So the decision framework is simpler than the tooling market wants you to believe. If you're small — a handful of services, modest spend — a single provider with good error handling and a documented manual-failover runbook is probably the right amount of engineering. Add a second vendor when you have a specific, named driver: a hard uptime SLA that a single provider's real-world reliability can't meet, a genuine capability gap between models, cost arbitrage at a volume where it actually moves the number, or a compliance requirement for data residency. "It felt safer" and "the microservice team does it" are not drivers. They are how you end up paying a permanent 20–30% tax to protect against a few hours of downtime a year.

Redundancy in stateless systems is close to free because the replicas are identical. Redundancy across model vendors is expensive because the models are not — and the entire value of a language model lives in the ways it is not identical to the others. Add the second vendor when the outage math demands it. Just go in knowing you're not buying a spare tire. You're adopting a second driver who takes a different route to the same destination, and you now have to make sure both of them know the way.