The Abstraction Layer That Made Every Model Mediocre

Somewhere in your codebase there is a gateway that lets you swap claude-sonnet for gpt-5 by changing one string. Your architecture review praised it. Your CTO sleeps better because of it. And it is quietly costing you 90% cache discounts, schema-enforced outputs, and the reasoning-effort knobs that separate a great production model from a mediocre one.

That is the unadvertised price of the unified LLM API. Every abstraction layer that promises "swap providers in one line" delivers that promise by projecting every provider onto the subset of features they all share — and the features that fall outside that subset are precisely the ones where providers compete hardest. Prompt caching semantics, structured output enforcement, extended thinking budgets, server-side tool execution: these are the levers that determine your real cost and quality curve, and your gateway may be silently dropping them on the floor.

The result is a strange equilibrium: teams adopt a routing layer to avoid betting on the wrong model, and in exchange they run every model in its least capable configuration. "We can switch anytime" becomes "we use every model at its worst."

The Lowest Common Denominator Is Lower Than You Think

The unified interface most gateways expose is some dialect of the OpenAI chat completions API, because that is the de facto lingua franca. Anything OpenAI-shaped passes through cleanly. Everything else has to be translated, emulated, or ignored.

Consider what actually differs across the three major API families:

• Prompt caching. Anthropic uses explicit cache_control markers with a 1.25× write premium and 90%-discounted reads; OpenAI caches automatically on matched prefixes with no write fee; Bedrock's Converse API has its own directive shape and TTL rules. These are not cosmetic differences — the directive shape, the response usage fields, and even when caching fires are all different. A gateway that normalizes requests into one canonical shape has to translate into three native dialects on the way out and reconcile three different usage objects on the way back. Many don't; they just strip the markers.
• Structured outputs. Every provider implements constrained decoding with a different parameter name, a different request shape, and different levels of actual schema enforcement. Some enforce JSON Schema at the decoder; some only guarantee "valid JSON, shape unspecified." A gateway that maps everything to response_format: json_object has downgraded your guaranteed-schema calls to hope-and-parse calls without telling you.
• Reasoning controls. Effort knobs, thinking budgets, and interleaved reasoning are named and shaped differently per provider, and some of them only exist on specific model generations. The difference between effort: low and effort: high on a hard task can dominate every other tuning decision you make — and it's exactly the kind of parameter a translation layer either passes through untyped or refuses to pass at all.

Each of these is worth more than the routing layer itself. Prompt caching alone can cut input cost by an order of magnitude on agentic workloads where the conversation prefix is replayed on every turn. If your gateway ate the cache markers, you paid full price on every call and the line item never told you why.

Portability Was Already a Myth Before the Gateway

The deeper problem is that the thing the gateway insures you against — being stuck on one provider — is not actually solved by API compatibility. The API is the cheapest part of a migration.

Prompts are model-specific artifacts. Research on cross-model prompt transfer calls this "model drifting": a prompt engineered against one model measurably underperforms when replayed against another, even a stronger one. In one benchmark, a prompt tuned for GPT-4o scored 92% on HumanEval when moved to o3 — a model capable of 98% on the same task with a prompt written for it. The prompt didn't break; it just quietly left six points on the table. Multiply that across every prompt in your system and "we swapped models over the weekend" becomes "we shipped a silent regression to every feature at once."

And prompts are only the first layer. Tool-call conventions differ. Refusal boundaries differ. Verbosity defaults differ — one model is "technically correct more often" while drowning users in nitpicks, which your evals won't catch unless they measure acceptability rather than raw accuracy. Migrating providers means re-running evals, re-tuning prompts, re-calibrating output parsers, and re-learning failure modes. The one-line model swap is real; it's also roughly 10% of the work.

So the honest accounting is this: the gateway gives you API portability, which you'd need maybe once every year or two, and charges you feature degradation on every single request in between. That's a bad trade unless you price it deliberately.

A Worked Example: The Agent Loop That Paid Full Price

Here is how the tax compounds in the workload that matters most right now — the agentic loop.

An agent conversation replays its entire prefix on every turn: system prompt, tool definitions, and the growing transcript all go back to the model each time it decides the next action. A 20-turn session with a 5,000-token system-and-tools block and a transcript growing by 1,000 tokens per turn resends the same prefix twenty times. With provider-native caching, turns 2 through 20 read that prefix at a 90% discount. Without it, you pay full freight on every replay — the input bill for the session lands somewhere between five and eight times higher, depending on transcript growth.

Now put a translation layer in the middle. If the gateway strips or mangles the cache markers — or normalizes the request in a way that shifts the token prefix between turns, which breaks prefix-matching caches just as thoroughly — the discount silently disappears. Nothing errors. The responses are identical. The only symptoms are a usage object with zero cache-read tokens and a bill that is several times larger than it needs to be, attributed to "agents are just expensive."

The same loop loses quality, not just money. Agent reliability lives in schema-enforced tool calls (a malformed argument at turn 7 derails everything after it) and in reasoning-effort control (planning steps deserve high effort, mechanical steps don't). Both are provider-native features. An agent running through a lowest-common-denominator interface is paying more per turn to fail more often — the worst point on the curve.

When the Tax Is Worth Paying

None of this means "delete your gateway." It means knowing which of its jobs justify the toll. The routing layer earns its place when:

• You genuinely run multiple models in production simultaneously — cheap model for classification, frontier model for generation, fallback chain for outages. Then the unified interface is load-bearing infrastructure, not insurance.
• You need centralized governance: per-team budgets, audit logs, PII redaction, key management. These are gateway-shaped problems, and solving them once beats solving them in every service.
• The calls flowing through it are commodity calls. Summarize this ticket, classify this email, extract these fields. For workloads where any competent model clears the bar, lowest-common-denominator is fine — that's what a commodity is.
• Failover matters more than peak quality. If a provider outage is an existential incident for your product, mid-tier output from a backup model beats no output.

The tax is not worth paying on your flagship path — the agent loop, the long-context RAG pipeline, the workload that dominates your bill. That's exactly where provider-native features pay: it's where caching saves real money, where schema enforcement prevents real parse failures, and where reasoning budgets move real quality metrics.

The Escape-Hatch Pattern

The practical architecture is not "gateway everywhere" or "native SDKs everywhere." It's a two-tier policy with an explicit boundary:

Tier 1 — commodity traffic goes through the gateway. High-volume, low-differentiation calls where portability and central governance are worth more than the last 10% of capability. Accept the common denominator here on purpose.

Tier 2 — flagship paths get provider-native access. Your core agent loop talks to the provider's own API shape — directly, or through the gateway's passthrough mode if it has one. The better gateways do: LiteLLM forwards unrecognized parameters straight to the provider and exposes native-format passthrough endpoints precisely because full translation is a losing game. Use those hatches deliberately rather than treating them as impurities.

Two rules make the pattern survivable. First, make the escape hatch visible in code review — a typed wrapper per provider-native feature (with_prompt_cache(...), with_schema(...)) rather than raw dictionaries smuggled through an extra_body field. When you eventually migrate, you grep for the wrappers and get an honest inventory of what's provider-coupled, instead of discovering it in production. Second, audit what the gateway drops. Log the request that leaves the gateway, not just the one that enters it, and diff. Check the usage objects for cache-read tokens actually appearing. Most teams that "enabled caching through the gateway" have never verified a single cached read came back.

And if insurance is the real goal, buy actual insurance: keep evals green against a second provider by running a small shadow-traffic slice against it monthly. That tells you your true switching cost — which the one-line swap never did.

Portability Is a Capability, Not an Architecture

The one-line model swap is a demo, not a strategy. Real portability lives in your eval suite, your prompt-migration playbook, and your inventory of provider-coupled features — none of which a unified API gives you, and all of which you can build while still using every model at full strength.

So flip the default question. Not "how do we stay portable?" but "which calls deserve the best each provider can do, and which calls are commodity?" Route the commodity, go native on the flagship, and make the boundary explicit enough that you always know what a migration would actually cost. The teams that get burned aren't the ones who coupled to a provider — they're the ones who paid the abstraction tax on every request for years and still discovered, on migration day, that the prompts were the hard part all along.