Eval Set Decay: Why Your Benchmark Becomes Misleading Six Months After You Build It

You spend three weeks curating a high-quality eval set. You write test cases that cover the edge cases your product manager worries about, sample real queries from beta users, and get a clean accuracy number that the team aligns on. Six months later, that number is still in the weekly dashboard. You just shipped a model update that looked great on evals. Users are filing tickets.

The problem isn't that the model regressed. The problem is that your eval set stopped representing reality months ago—and nobody noticed.

This failure mode has a name: eval set decay. It happens to almost every production AI team, and it's almost never caught until the damage is visible in user behavior.

What Eval Set Decay Actually Looks Like

An eval set is a frozen snapshot of the user distribution at the moment you built it. Real traffic is not frozen. Users evolve how they phrase requests. New use cases emerge. Edge cases that didn't exist in your beta cohort become common in your broad-release cohort. Your product shifts in positioning, which attracts a different user profile with different expectations.

The divergence between your static eval set and your live distribution is not random—it's directional and accelerating. Here's what it produces:

Misleading green metrics. Your eval suite passes at 91%. Real users are frustrated. The difference is that your eval cases are drawn from six-month-old query patterns, and current traffic has evolved in ways your eval doesn't test. You are measuring performance on questions users used to ask.

Invisible failure class emergence. Production accumulates new failure modes over time—phrasing patterns, request formats, edge cases—that your eval set predates. When you ask "did this model update cause a regression?" your eval can only compare against the failure modes it knew about when you built it. Novel failures are invisible.

Benchmark saturation without capability gain. When a team keeps iterating their model against the same eval set, they eventually overfit to it. Scores climb toward 98-99%. Real performance plateaus or falls. The eval is no longer discriminating—it is being gamed, even without anyone trying to game it.

Research on synthetic benchmarks finds they catch only 60-70% of failures that appear in actual user behavior. That gap widens as the eval set ages.

The Mechanism: Why User Behavior Drifts

Understanding why drift happens helps you predict where it will happen fastest.

Vocabulary and phrasing evolution. Users adapt their language to what they observe works. If early adopters learned that certain phrasings produce better responses, they share those patterns. Later users adopt them. The phrasing distribution shifts from what you saw in beta. Your eval cases, written in the old idiom, stop being representative.

Cohort composition change. A product launched to ML engineers has a completely different query distribution than the same product at 10x scale with enterprise operations teams as the primary user. Your eval set built for the first cohort doesn't represent the second one.

Use case evolution. Users discover unanticipated applications for the product. These applications involve different request formats, domain vocabulary, and success criteria than your original design anticipated. The eval set has no coverage of them.

Domain knowledge currency. In anything that touches current events—pricing, regulations, technical specifications, personnel—the ground truth in your eval cases can become wrong simply through the passage of time. You are now testing whether the model agrees with outdated facts.

The decay velocity varies by domain. High-churn domains like financial news, software tooling, and current events can make an eval set stale in weeks. Stable domains like basic math reasoning or general document summarization decay more slowly—but they still decay.

Measuring Decay: How to Tell When Your Eval Is Lying

The first step is to make the drift visible. Most teams only realize their eval has decayed after a production incident. There are earlier signals.

Production-eval semantic distance. Embed a sample of recent production queries and a sample of your eval cases in the same embedding space. Measure the centroid distance and the overlap between the two distributions. If the embedding spaces are diverging, your eval is covering different territory than your traffic. This can be automated as a weekly check.

Coverage heatmaps. Cluster your production queries by topic, intent, and phrasing pattern. Measure what fraction of each cluster has corresponding eval cases. You will find entire regions of live traffic with zero eval coverage—these are your blind spots. Coverage heatmaps make the decay spatial instead of abstract: you can point to the specific query types you cannot currently measure.

Diversity metrics over time. Track n-gram entropy and semantic diversity of your incoming traffic versus your eval set. When the traffic diversity measure diverges upward from your eval diversity measure, users are exploring territory your eval doesn't cover. Vendi Score and embedding dissimilarity metrics are useful here because they don't require labeling—they operate on the query distribution directly.

Statistical drift tests. Apply Kolmogorov-Smirnov or Population Stability Index tests to the embedded query distributions on a rolling basis. PSI values above roughly 0.2 indicate actionable drift that warrants investigation. These tests are borrowed from traditional ML monitoring but apply directly to LLM input distributions.

A customer support bot with 99.2% uptime and healthy latency can have its hallucination detection score drop from 94% to 82% over a quarter with no infrastructure alerts firing. Quality drift of this kind is invisible to infrastructure monitoring—you only catch it if you are running quality-based evals against fresh samples of production traffic.

The Fix: Rolling Eval Maintenance

Treating eval sets as permanent artifacts is the source of the problem. The fix is to treat eval maintenance as an ongoing engineering function, not a one-time task.

Continuous production sampling. The most reliable source of representative eval cases is production traffic. Establish a pipeline that samples live queries at a configurable rate (typically 5-10% of traffic), routes them through quality filters, and feeds them into a candidate eval pool. From this pool, a combination of automated and lightweight human review selects cases to promote into the active eval set. The key property: your eval distribution is continuously refreshed from the live distribution.

Sample prioritization matters. You don't want to add eval cases that look like your existing cases—that wastes capacity and inflates coverage without reducing blind spots. Weight new sample selection toward: (a) query types with low existing coverage, (b) cases where the current model shows high uncertainty, and (c) failure cases surfaced from production feedback signals.

Rolling retirement of stale cases. Adding new cases without removing old ones inflates the eval set without improving its representativeness. Audit your eval cases for age and recency relevance. Cases from deprecated product flows, outdated topic areas, or pre-refactor prompt formats should be retired. A useful heuristic: any case whose ground truth depends on knowledge or product state from more than six months ago should be reviewed for retirement.

Structured refresh cadence. Rather than ad-hoc updates, establish a scheduled eval review cycle. For most teams, a monthly review covers the majority of decay risk without becoming a maintenance burden. Higher-velocity domains may need biweekly cycles. The review should answer: (a) what failure types appeared in production last month that our eval doesn't cover, (b) what regions of traffic have low eval coverage, and (c) which existing cases are stale.

The temporal distribution shift is predictable: if you can measure how far your eval distribution has drifted from live traffic, you can project forward when it will reach an unacceptable divergence threshold. Set that threshold explicitly rather than waiting for user complaints to define it for you.

Eval set versioning. Your eval set is a dependency of your model evaluation pipeline. It should be versioned the same way your code is versioned. When you publish a model quality number, the eval version should be pinned alongside it. This makes it possible to answer questions like "did quality actually improve, or did we just update the eval?" and "what was the live-traffic coverage of the eval we ran last quarter?"

Eval set versioning also enables retrospective analysis. When a production incident occurs, you can rerun the model against the eval set version that was active at the time of the incident and compare against the current eval set. If the incident-era eval shows no regression, that is evidence the eval had blind spots—not evidence the model was fine.

What Not to Do

A few patterns that teams adopt thinking they address eval decay, but don't:

Expanding the eval set without diversifying it. Adding 500 new cases that are semantically similar to existing cases doesn't improve coverage. Diversity metrics should be used to measure the incremental value of each new batch of cases before adding them.

Relying solely on LLM-as-judge evaluation on static cases. If both the judge prompt and the eval cases are static, you've introduced a second decay vector: the judge itself may be evaluating against outdated criteria as the product's definition of quality evolves. Refresh judge rubrics when you refresh eval cases.

Trusting A/B test metrics as a proxy for eval coverage. A/B tests measure aggregate behavioral signals over the live distribution, but they are lagging, noisy, and don't provide case-level insight into where the model fails. They're complements to eval maintenance, not substitutes.

Treating decay as a model problem. The framing "our eval is fine, the model is behaving unexpectedly" is almost always backwards. The model is behaving on real traffic. The eval is not measuring real traffic. Start with the assumption that the eval is wrong.

The Organizational Challenge

Eval maintenance is unglamorous. It doesn't ship features. It doesn't improve model scores. It produces costs in the form of labeling time and infrastructure without producing an obvious output artifact. As a result, it is almost universally underfunded until something breaks in production.

The counterargument is straightforward: the cost of running on a stale eval set is model quality investments that are impossible to measure, regressions that go undetected for weeks, and production incidents that could have been caught in staging. Eval maintenance is insurance, and like most insurance, it looks expensive until you need it.

Teams that do this well treat eval set freshness as a first-class quality metric, reported alongside model accuracy. When the coverage heatmap shows a region of live traffic with no eval coverage, that gap gets a ticket. When the production-eval semantic distance exceeds a threshold, an alert fires. The eval set is treated as a living system, not a completed artifact.

The teams that don't do this will keep hitting the same failure mode: confident green eval scores on benchmarks that stopped representing production six months ago.

---

The practical starting point: take your current eval set, embed it alongside a week of production queries, and render a coverage heatmap. The blind spots will be immediately obvious. The question is whether you build a process to close them, or whether you wait to discover them in production.