Your Eval Suite Is Overfit to Your Incumbent Model

A new frontier model ships. It's cheaper, faster, and tops every public leaderboard. You run it through your eval suite — the one you've spent eighteen months building — and it scores worse than the model you're running today. So you keep the incumbent, file the challenger under "not ready," and move on.

Here's the uncomfortable part: that result tells you almost nothing about which model is better. It tells you that your eval suite was built by watching your current model fail, one production incident at a time, and then patched to make those specific failures go away. The suite isn't a neutral measurement of quality. It's a catalog of one model's scar tissue. And a challenger that has different weaknesses will always look worse against a test set assembled from the incumbent's particular weaknesses — even when it's better on the traffic you actually serve.

This is incumbent bias, and it's the switching cost nobody prices into the migration decision. It quietly locks you onto a model long after a better option exists, and it does it while wearing the costume of rigorous engineering.

Eval Suites Grow by Autopsy

Nobody sits down and designs a comprehensive eval suite up front. That's not how they get built. They accrete.

The lifecycle is always the same. A model ships something wrong in production — it hallucinates a refund policy, drops a required JSON field, ignores a negation, mangles a date. Someone gets paged. The trace gets diagnosed. And the fix is almost never "retrain the model" — it's a prompt patch, a few-shot example, a guardrail, a post-processor. Then, if the team is disciplined, that failure becomes a permanent eval case so the same regression can't sneak back in.

Do this a few hundred times and you have a mature eval suite. It looks robust. It has hundreds of cases, each one earned through real pain. But look at where every case came from: each row exists because your current model failed it at least once. The suite is a photographic negative of one specific model's weak spots. It's not a map of the task — it's a map of the incumbent's history of screwing up the task.

The prompt fixes make it worse. When you patch a failure with a few-shot example or a carefully worded instruction, you're tuning the prompt to the model. Those two are now co-adapted. Your instructions lean on quirks of how this particular model tokenizes, where it pays attention, what phrasing it responds to. The eval case and its fix are a matched set, filed to the exact contour of one model's behavior.

Swap in a challenger and both halves of that matched set misfire. The challenger doesn't have the incumbent's failure mode, so the case was never testing anything it struggles with. And the prompt fix was shaped around the incumbent's quirks, so it may actively confuse the new model. You're not measuring the challenger. You're measuring how well the challenger imitates the model you already have.

Why "Worse" Usually Means "Different"

Run a genuinely capable challenger through an incumbent-shaped suite and you'll typically see a distinctive signature: it fixes a bunch of cases the incumbent never failed (so they're not in your suite and you get no credit) while failing a different, smaller set of cases (which are conspicuously absent from your regression tests, so those failures go undetected until production).

Aggregate that into a single pass rate and the challenger lands a few points below the incumbent. The number reads as "worse." But decompose it and the real story is "differently distributed errors" — and possibly fewer errors on the traffic you actually see. The suite can't show you that, because it only contains failures the incumbent already taught it about.

There's an empirical analog in the benchmark world. When Scale AI built a fresh parallel version of the GSM8K grade-school math benchmark, several model families dropped by double-digit accuracy on the new problems versus the original ones — a textbook overfitting signature. The models hadn't memorized the specific items, but the ecosystem had implicitly tuned toward that fixed set. Your private eval suite is far more overfit than GSM8K ever was, because it wasn't just tuned to a fixed set — it was generated by a single model's failures, with fixes co-designed against that same model.

This is Goodhart's Law with a subtle twist. The usual framing is that when a measure becomes a target, it stops being a good measure. The incumbent-bias version is quieter: the measure was constructed from the target's mistakes, so it was never a neutral measure to begin with. Passing it means "behaves like the incumbent," not "is good at the task." Any model that behaves differently — including a better one — gets penalized for the difference.

Separating Regression Cases from Capability Cases

The fix isn't to throw out your hard-won eval suite. Those production failures are real, and you genuinely don't want them recurring. The fix is to stop treating one pile of cases as if it measured one thing. It measures two, and they need to be scored separately.

Model-specific regression cases exist to prevent a known failure from recurring. They're inherently tied to a model's behavior. "Given this ambiguous refund request, model X used to invent a policy — make sure it doesn't anymore." That's a real safeguard, but it's a statement about model X. A challenger that never had that failure will pass it trivially, and passing it proves nothing about the challenger's quality. These cases are for guarding a specific model in production, not for comparing two candidates.

Model-agnostic capability cases describe what the task requires, independent of who's performing it. "Extract the delivery date from this email; the correct answer is 2026-03-14." That's true no matter which model reads it. These cases measure the task, not a model's history with the task. They're the only part of your suite that can fairly compare candidates.

Most teams have never drawn this line, so both kinds live in one folder and get averaged into one number. Draw it explicitly. Tag every case as regression (bound to a model, prompt fix attached) or capability (bound to the task, defined by ground truth). When you evaluate a challenger, score it on the capability set and read the regression set as diagnostic, not as pass/fail — a challenger failing an incumbent-specific regression case is information about where it differs, not evidence that it's worse.

Here's a useful tell for which bucket a case belongs in: if the "correct answer" was authored by looking at what a good response should be, it's a capability case. If it was authored by looking at what the incumbent did wrong and encoding "don't do that," it's a regression case. Two different provenances, two different jobs.

A Vendor-Swap Protocol That Doesn't Cheat

Once you accept that your standing suite is incumbent-shaped, the evaluation protocol for a model swap has to change. The goal is to score the challenger on the distribution you actually serve, using cases neither model has been tuned against.

Draw fresh task samples from live traffic. Pull a new, unseen sample of real production inputs — ones that postdate your suite and were never used to author a prompt fix. Label them independently: from ground truth where it exists, or from a judge that isn't the author of your existing fixes. This is your real test set. Freeze it before anyone looks at model outputs, so the challenger and incumbent are graded on identical, uncontaminated inputs.

Neutralize the prompt. Your production prompt is co-adapted to the incumbent — its few-shot examples, its defensive instructions, its phrasing all lean on quirks of the current model. Grading a challenger through that prompt measures compatibility with the incumbent's crutches, not capability. Give the challenger a fair shot at its own best prompt for the same task before you conclude it's worse. Sometimes a single prompt rewrite erases the entire gap, because the "gap" was never in the model.

Score error distributions, not just the aggregate. Don't stop at a single pass rate. Bucket errors by type and severity. A challenger that trades ten low-severity formatting misses for two fewer catastrophic policy hallucinations is a win your aggregate number will call a loss. Migration decisions live in the distribution, not the mean.

Then A/B in production. Offline evals identify promising candidates; they don't settle the question, because non-deterministic outputs are exquisitely sensitive to prompt, parameters, and version. Route a slice of real traffic to the challenger and measure outcomes users actually feel — task completion, escalation rate, downstream corrections — not proxy pass rates on a frozen set. This is the only stage that isn't shaped by anyone's history with either model.

Migrate your regression cases too. Once you commit to a new model, don't drag the old model's regression cases forward as pass/fail gates. Re-baseline. Run the challenger in production, harvest its failures, and grow a new regression layer around the model you're actually shipping. Otherwise you spend the next year failing your new model against the old one's scar tissue — and you've just recreated the exact bias, one migration later.

The Meta-Lesson: Evals Have a Half-Life

The deeper point is that an eval suite is not a fixed asset that only appreciates. The regression half of it depreciates the moment you change models, because those cases were always claims about a specific model. Treat that half like a model-versioned artifact: pinned to the model that generated it, re-based when you migrate, never mistaken for a neutral yardstick.

The capability half is the durable part — that's your real institutional knowledge about the task, and it's the only fair basis for comparing candidates. Most teams have far less of it than they think, precisely because the accretion-by-autopsy process manufactures regression cases far faster than capability cases. Autopsies are cheap and automatic; articulating what the task actually requires is deliberate work almost nobody schedules.

Incumbent bias in evals is a real switching cost, and like most switching costs it's invisible until you try to switch. You feel it as "the new model just isn't as good," backed by a suite you trust because you built it honestly. But honesty in construction doesn't buy neutrality in measurement. A suite assembled from one model's mistakes will always flatter that model and slander its replacements. The way out isn't to trust the suite less — it's to know which half you're reading, and to test challengers on the task you serve rather than the incumbent's old scar tissue.