You Test on the Cheap Model and Ship on the Expensive One

Somewhere in your codebase there is a config file that says something like model: small-and-cheap under the test profile and model: frontier under production. It felt responsible when someone added it — why burn frontier-model tokens on CI runs that fire twenty times a day? But that one line quietly repealed a rule your team has followed for fifteen years without thinking about it: the environment you test in should behave like the environment you ship to.

The twelve-factor methodology called this dev/prod parity, and we got so good at it that we stopped noticing it. Docker gave us bit-identical runtimes. Infrastructure-as-code gave us identical topology. Then we put a language model in the middle of the request path and reintroduced the exact gap we spent a decade closing — except this time the divergent component isn't a database version. It's the part of the system that makes the decisions.

The model is not a dependency — it's the environment

When your staging database is Postgres 15 and production is Postgres 16, you worry a little. When your test suite exercises a small model and production runs a frontier one, you should worry a lot more, because the difference isn't an edge case in query planning. It's a different distribution over every output your system produces.

Two models that score within a few points of each other on a benchmark can diverge wildly on the traffic that matters: how they handle ambiguity, how often they refuse, whether they wrap JSON in markdown fences, how many tool calls they spend on the same task. Cross-model prompt-transfer research keeps finding the same thing — transfer gaps show up across model families and across coding, agentic, and planning tasks, because every prompt tweak you made was implicitly encoding one model's interpretation style. Your prompts didn't converge on "correct." They converged on that model.

This is why "the model" belongs in the environment column of your mental twelve-factor table, not the dependency column. A dependency has a version and a changelog. An environment is the thing your software's observed behavior is conditioned on. Swap it, and every green checkmark you've accumulated is evidence about a system you're no longer running.

The two directions of the mismatch, and why both bite

Teams split the tiers in two directions, and each has its own failure signature.

Test cheap, ship expensive. This is the common one: CI, local development, and pre-merge evals run on the small model to control spend; production gets the frontier model because users deserve the best. The failure mode is subtle because the production model is better — so teams assume the gap can only help them. It can't, for two reasons.

First, "better" is not monotone across behaviors. Frontier reasoning models take instructions more literally, refuse more readily, and surface edge cases that a smaller model rounded off. A prompt that worked because the cheap model glossed over an ambiguity may behave differently when the expensive model takes it at its word. Second, your evals stop measuring production at all. A green eval run on the cheap tier tells you the cheap tier still works. Nobody deployed the cheap tier.

Test expensive, ship cheap. The mirror image: prompts are developed and demoed against the frontier model, then production routes most traffic to a small model for cost. Here the failure is capability cliff rather than behavior drift. Small models under-call tools — they can execute a tool call fine but fail to recognize that they need one. The demo worked because the frontier model papered over a vague tool description with reasoning; the small model needs the description to be explicit, and nobody discovered that because nobody ran the real production path until launch.

Both directions share a root cause: the tier split was made by the finance instinct ("don't waste tokens") and never revisited by the engineering instinct ("what does this do to my test validity?").

Your eval suite is green on a model you don't run

The most expensive version of this problem hides in evaluation pipelines. Teams invest heavily in evals — curated datasets, regression cases for past bugs, CI gates that block deploys below a quality threshold. Then, to keep the pipeline affordable, they point the whole thing at the cheap tier.

Think about what that gate now certifies. A pull request that changes a prompt gets tested against the small model. If it passes, it merges, and the changed prompt goes to production — where a different model interprets it. The eval gate has become a ritual: it produces the feeling of regression coverage while measuring a system that doesn't serve traffic. Worse, it can actively mislead. A prompt change that regresses on the small model but would have been fine on the frontier model gets reverted; a change that's fine on the small model but regresses on the frontier one ships. You're steering by a compass calibrated to a different magnetic field.

Loading…