143 posts tagged with "evals"

The eval set that scored 92% last quarter is now scoring 94%, and the team is calling that progress. It isn't. The labels in that eval set were written against a rubric the annotators no longer hold in their heads. The product the model is being graded on has moved. The standards have moved. The annotators' own calibration has moved. What looks like a two-point improvement is the silent gap between a frozen artifact and a living product, and that gap widens every week the team doesn't refresh.

Annotation drift is the quiet failure mode of mature LLM eval programs. It doesn't show up as a regression — regressions are the easy case, because the number goes down and somebody investigates. It shows up as a number that stays green while the thing it's supposed to measure decays underneath it. Teams that have already built an eval set, written a rubric, and recruited annotators are the most exposed, because they trust the system they built and stop auditing the foundation.

Here is the awkward truth most AI teams discover six months too late: a single well-designed eval case routinely costs more engineering effort than the feature it is supposed to test. A prompt edit takes an afternoon. The eval case that gives you confidence the prompt edit didn't break something takes a domain expert two days of labeling, a calibration loop with a judge prompt, and a discussion about what "correct" even means for this user surface. The feature ships in a sprint. The eval that lets you ship the next ten features safely takes a quarter to mature.

The asymmetry isn't a bug. It is the structural shape of evaluation work. Labeling, edge-case curation, judge calibration, and rubric design are upfront fixed costs that don't scale with how many features you ship — they scale with how many distinct behaviors you want to verify. Meanwhile the feature side keeps producing what feels like cheap marginal output: "another prompt iteration," "one more tool added to the agent," "swap the model." Each looks individually small. Each silently increases the surface area the eval set must cover.

The demo went perfectly. The room clapped. Two weeks later, the same feature lands in the company Slack for internal use, and by Wednesday a senior engineer is posting screenshots with the caption "did anyone test this?" By Friday the channel has gone quiet — not because the bugs were fixed, but because the people who would have flagged them gave up and went back to their old workflow. The launch is still on the calendar. Nobody has cancelled it. Nobody has the political capital to.

This is the demo-to-dogfood gap, and the MIT NANDA initiative measured it last year at 95% — that is the share of enterprise generative AI pilots that produced no measurable P&L impact, and almost all of them had a demo somebody loved. The model was not the problem. The gap between the demo and the first week of internal use was the problem, and every team that has shipped an AI feature has watched some version of it play out.

An executive sends a one-line email: "great news — we're adding vision next sprint." The product manager interprets it as a one-week project: swap the model, expose an image parameter, ship. The eval team reads the same email and starts mentally drafting a four-week schedule that nobody has approved yet. By Friday, the disconnect surfaces in standup as a vague "we'll need to do some eval work" and everyone agrees to figure it out later.

That gap between "we added vision" and "we can safely ship vision" is the eval backfill tax. It is the work that quietly falls on the eval team every time a new model capability lands — multimodal input, tool use, longer context, reasoning traces, computer use — because the historical test cases were constructed in a regime where the model could not fail in the new ways the new capability introduces. The suite stays green, the headline benchmark goes up, and the production launch surfaces failure modes nobody wrote a test for.

A team I worked with recently lost their senior ML engineer. Two weeks later, the eval suite was still green on every PR — 847 cases, all passing, judge agreement at 92%. Six weeks later, a customer found a regression that should have been caught by the very first eval case in the support-quality bucket. When the team went to debug, nobody could explain why that case had been written, what failure mode it was supposed to catch, or why the judge prompt graded it on a 1–4 scale instead of binary. The case was still passing. It just wasn't testing anything anyone could name.

This is the eval bus factor: the silent failure mode where the person who decided what "correct" means for your AI feature was also the person who curated the test cases, calibrated the judge, and absorbed every implicit labeling tradeoff in their head. When they leave, the suite remains green but stops generating reliable promote/reject signal — because nobody else can extend it, debug a flaky judge, or evaluate whether a new failure mode belongs in the test set.

A healthy eval set is supposed to be a sign of maturity. It is also, on any given Monday, a thousand failed cases sitting in a queue with a human reviewer who has eight hours and a per-case throughput of about fifty. The arithmetic is brutal: roughly one in twenty failures gets read. The other nineteen wait. Which nineteen wait, and which one gets the seat, is decided by whichever order the file happens to load in.

Most teams call this "reviewing failures." It is closer to a lottery weighted by alphabetical order. A failure case that affects two percent of production traffic and lives at the top of the file gets attention. A failure case that affects forty percent of production traffic and lives near the bottom gets a glance on Friday afternoon, if at all. The team ships a fix for the small problem on Tuesday and writes a retro on Thursday wondering why the dashboard hasn't moved.

Open a mature production system prompt and search for the word "not." On a feature that has shipped through three quarters and survived a handful of incidents, you will almost always find a section that looks like a list of regrets — "do not give medical advice, do not generate code matching these patterns, do not produce content with this regex, do not impersonate these competitors, do not use these phrases." Each line traces back to a specific incident. Each line was added with confidence by an engineer who said "this will fix it." And the list grows, every quarter, in the same way a museum acquires exhibits.

What very few teams will admit out loud is that this list — the prompt's most defensive, most carefully reviewed section — is also the most useful artifact in the entire feature for the wrong reader. A determined user who extracts the system prompt now has a curated, organized, model-readable inventory of every behavior the team is afraid of. The forbidden list is a recipe. The team wrote the cookbook.

A team ships a self-critique loop into production because the benchmark numbers looked irresistible: Self-Refine reported a 20 percent absolute improvement averaged across seven tasks, Chain-of-Verification cut hallucinations by 50 to 70 percent on QA workloads, and reflection prompts pushed math-equation accuracy up 34.7 percent in one widely-cited paper. A month later the finance review surfaces the bill. The product's per-request cost has roughly tripled, p99 latency is up by a factor of three, and the actual quality lift that survived contact with production traffic is closer to three percent than thirty. The self-critique loop is doing exactly what it advertised. The team just never priced it.

This is the self-critique tax: a reliability pattern that reads like a free quality win on a slide and reads like a structural cost increase on an invoice. The pattern itself is sound — there are real cases where generate-then-verify is the right answer. The failure mode is shipping it as a default instead of as a calibrated intervention, and discovering at the wrong time of the quarter that "the model checks its own work" was actually a procurement decision.

Six months of green CI is hiding the fact that roughly forty percent of your eval set no longer represents what users actually do with your product. The suite still runs. The judge still scores. The dashboards still glow. But the cases were written against a query distribution, a corpus, a tool surface, and a regulatory text that have all moved underneath them — and a green run now means "yesterday's product still works on yesterday's reality," which is not the question you are paying CI to answer.

This is snapshot eval decay, and it is the slowest, most expensive failure mode in AI evaluation. Slow because the suite never fails — staleness shows up as inability to discriminate between models, not as red builds. Expensive because by the time someone notices that a model swap which the evals approved caused a production regression, the team has already accumulated a year of "we ship when evals pass" muscle memory built on top of an asset that quietly stopped working.

A user's calibration of an AI feature is one of the most expensive things you ship. It costs them weeks of attention: learning which prompts work, where the model's reliable, when to double-check, what to ignore entirely. Then a single visible failure — a wrong number in a generated report, a hallucinated citation the user pasted into a deck, a confidently-incorrect recommendation they acted on — can vaporize all of it in one session. The recovery curve isn't symmetric. The user's prior was "this is reliable," and the update doesn't land as a data point. It lands as a betrayal.

The team measuring DAU sees nothing for weeks. The user keeps opening the app out of habit, runs a few queries, doesn't act on the output, and then quietly stops. By the time engagement metrics flinch, the trust event that caused it is two months old and nobody on the team remembers shipping it.

A senior engineer walks into the promotion calibration meeting. They shipped a fine-tuned reranker that lifted retrieval quality eight points. They built the eval harness that turned a two-week QA cycle into a one-hour CI gate. They authored the prompt change that drove a two-point conversion lift. By any reasonable measure, they had a defining year.

They don't get promoted. The packet, as written, reads like "I tuned some numbers." The colleague next to them — who shipped a CRUD feature behind a launch banner with QPS, latency, and a Friday demo — gets the nod instead. The committee is not malicious. It is using a vocabulary it has, applied to a packet that didn't translate the work into that vocabulary.

This failure mode is now common enough to be a pattern. AI engineering work doesn't decompose cleanly into the artifacts that calibration committees were trained to evaluate. The packet template was written for deterministic systems shipped in deterministic ways, and the engineers who do the most leveraged work in the AI stack are paying the tax.

There is a quiet failure mode in production-grade LLM evaluation that no leaderboard catches: your test set is built from the users who stayed, so it never asks the questions that made the others leave. Quarter over quarter the eval scores climb, the dashboards turn green, and net retention sags anyway. The team chases "is the eval gameable?" when the real story is simpler and harder. The eval distribution drifted toward survivors, and survivors are exactly the population whose feedback you least need.

This is the WWII bomber armor problem in a new costume. Abraham Wald looked at returning planes, noticed where the bullet holes clustered, and pointed out that the holes you should reinforce against are the ones on planes that didn't come back. Replace bombers with users, replace bullet holes with failed turns, and you have the central pathology of eval sets seeded from production traces.