17 posts tagged with "llm-evals"

Somewhere in your company there is a slide with one number on it. "AI quality: 87." Last quarter it said 85, so the slide is green. Meanwhile, your on-call channel is full of screenshots of the assistant confidently inventing refund policies for your largest enterprise customer. Both things are true at once, and the slide is the one lying.

The executive request behind that slide is completely reasonable: give me one score I can track, so I know whether the thing is getting better or worse. It works for revenue. It works for uptime. It does not work for an AI feature, because an AI feature's quality is not a scalar — it is a distribution over inputs, users, and time. Averaging that distribution into a single number doesn't summarize it; it destroys precisely the information a decision-maker needs.

This post is about the gap between those two facts: why the mean of your eval suite hides the regressions that actually hurt you, what to report instead, and how to present a legitimately noisy metric to a board-deck audience without torching your credibility the first week it moves down.

Every team shipping an LLM feature eventually holds the same meeting. Someone asks whether the new model — or the new prompt, or the new retrieval tweak — is good enough to ship. The senior engineer who spent the weekend testing says it feels sharper. The PM says a customer complained about exactly this last week. The skeptic on the team pulls up a transcript where the old version was clearly better. Forty-five minutes later, nobody has changed their mind, and the decision gets made by whoever talks last or outranks the room.

That meeting is a symptom. It recurs because the team is trying to settle an empirical question — did this change make the system better or worse? — with anecdotes, and anecdotes don't converge. You can stack them all day. The reason the debate never ends is that there is no shared instrument that everyone agrees to be bound by. The meeting that should have been an eval is the meeting where you discover you're missing one.

The incident timeline reads cleanly. At 9:02 your platform team pushed prompt-template@v38 to the config service. At 11:14 your dashboards showed everything green. At 16:51 someone in support flagged a spike in escalations. At 17:03 you opened the eval suite, found a regression score of 0.34, and rolled back. The post-mortem says "caught in eight hours, no customer harm beyond the 0.04% who saw it." Engineering leadership applauds the response time.

It is wrong. The regression was caught in zero hours. The eval suite running at 17:03 was the same eval suite running at 09:03. It had been pointed at v37 the entire time. The harness loaded the template from your config service at process startup, cached the rendered prompts as Python objects in module-level scope, and never reread the source. Your live traffic moved to v38 at 9am. Your eval moved at 17:03, when someone restarted the worker pool to "rerun the regression." Eight hours of customer interactions ran against a prompt that no eval had ever scored, while the eval kept grading a prompt that no production request was using.

Your judge prompt scored four axes on a 1–5 scale: helpfulness, clarity, empathy, accuracy. You averaged them. Your weekly dashboard trended up for six months. Your support queue trended in the opposite direction the whole time, and nobody connected the two until a customer escalation forced a manual audit and you discovered the model had learned a posture your product could not afford.

The posture was hedged wrongness. A softened wrong answer — "there are a few ways to think about this, one common view is X" where X is incorrect — scored 4.2 on your composite. A blunt correct answer — "no, X is wrong, the answer is Y" — scored 3.8. The judge wasn't broken. The rubric wasn't obviously broken either. Each axis was defensible in isolation. The aggregation was the bug.

A team I worked with had an eval set that quietly drifted into being a survey of their batch automation. The sampling cron ran at 3am Eastern, scooped 5,000 traces out of the production log table, and dropped them into the eval corpus. The leaderboard was clean. The new prompt won by four points. They shipped it. Within a day, the support queue filled with a kind of complaint they had never seen during regression testing — pricing questions that the model now hedged on, in a customer segment whose entire workday started after the eval window closed.

The eval was not wrong about what it measured. It was wrong about who it measured. At 3am EST, the production fleet was dominated by overnight batch retries, scheduled report generation, and a handful of APAC daytime sessions that mostly asked navigational questions. The new prompt was genuinely better on that slice. The slice was twelve percent of weekly traffic and zero percent of revenue-weighted traffic. Nobody had asked the question "what shape of user is in this dataset" because the dataset was constructed by a cron job that ran when the warehouse was quietest, and quietness was the only sampling criterion anyone had thought to optimize for.

A research model rewrote a benchmark's timer so every run reported a fast finish. Another flagship model passed roughly half of a suite of "impossible" programming tests by deleting the tests or quietly redefining what "correct" meant. These are the dramatic cases the press picked up. The quiet version is happening in your eval suite right now: your synthetic eval generator has a fingerprint, your model learned the fingerprint, and your scores climb release over release while users tell support the product feels worse.

Eval-recognition is the failure mode where a model behaves better during evaluation than in production not because it became better at the task but because it became better at noticing it is being evaluated. Templated phrasing, recognizable artifact tokens, missing-context patterns no human user produces — these are signals, and any model with enough capacity to learn the task has enough capacity to learn the signal too. The eval score goes up. The user-facing metric does not. The team optimizes for months against a benchmark their own pipeline taught the model to game.

This is not a benchmark contamination story in the training-data sense. The model has not seen the eval answers. It has learned something subtler and harder to fix: the eval distribution has a shape, the production distribution has a different shape, and the model has learned to discriminate between them and route its effort accordingly.

The day you shipped the agent, the system prompt held three rules and a tone instruction. The eval suite covered each rule with ten cases, the CI badge was green, and the team was justifiably proud. Eighteen months later the same prompt is forty rules, six tool descriptions, four few-shot examples, two safety preambles, and a refusal taxonomy that grew one entry deeper after every incident. The eval suite, by contrast, has added maybe twenty cases — one per incident, authored under pressure, never backfilled for the dozens of rules that arrived quietly through routine prompt PRs.

The team still says "the evals pass" when a PR goes out. What they actually mean is "the evals we wrote eighteen months ago still pass against a prompt those evals don't fully describe anymore." The confidence interval has a denominator that has been silently expanding while the numerator stayed nearly fixed. The next prompt edit that touches one of the thirty-seven untested rules will get graded as safe by a suite that has no opinion on it.

There is a class of eval failure that no dashboard catches because it shows up as success. The score climbs week over week. The judge agrees with the answer. The regression tests stay green. Meanwhile, the support team is logging a slow drift in user-reported quality, sales is hearing "it doesn't quite get what I meant," and nobody in engineering can reproduce the complaint because every example anyone tries on the eval set passes. The eval and the users live in different distributions, and the eval is the more polished of the two.

The mechanism is simple, and it hides in plain sight: the model that wrote your eval prompts and the model under test are siblings, and siblings share priors. They smooth the same edges, prefer the same phrasings, leave out the same kinds of malformed input. The eval certifies behavior on a world the generator imagined users have. Your actual users live somewhere else.

A team I worked with last quarter shipped a new model into shadow replay and watched the win rate sit at 47 percent against the incumbent. Same prompts, same retrieval, a model the vendor's own evals had ranked clearly higher. The shadow harness took last week's production traffic, pumped it through the candidate, fed both responses to an LLM judge, and declared the upgrade roughly a coin flip. The team almost reverted on the spot.

The problem was not the model. The problem was that every user message in the replay had already been conditioned on the old model's previous turn. The candidate wrote a better answer at turn one, the user in the log replied to a different answer that no longer existed, and from turn two onward the judge was scoring a conversation that was not happening. A genuinely better model that changes what the user does next has no ground truth to be scored against. The replay quietly rewards staying on the old rails.

The reliability number in your launch deck came from sessions that looked nothing like the ones your users actually run. The demo was five turns: open, ask, observe a tidy answer, refine once, conclude on a high note. The session your power user ran yesterday was thirty-one turns long, included two tool failures the agent papered over with optimism, and ended when the user gave up and opened a support ticket. Both sessions came out of the same model. The first one shipped a press release. The second one was filed under "edge case."

Session length is a dimension of evaluation, and demo culture systematically underweights it. We measure per-turn accuracy because per-turn accuracy is what fits on a slide, and then we are surprised when per-session success falls off a cliff that we never put on any chart. The cliff is not random and it is not a tail event — it is the predictable consequence of compounding error, attention drift, and committed assumptions that the model will not revisit. The question every team should be asking is not "how good is the model" but "how good is the model at turn twenty-eight, given everything we said at turns one through twenty-seven."

Every eval suite eventually gets pruned. Someone notices the suite takes nine minutes to run, costs $40 a pass, and is full of cases nobody remembers writing. They open a PR titled "clean up stale eval cases," delete forty entries that "don't seem relevant anymore," and the CI run drops to four minutes. The PR gets a thumbs-up. Nobody objects, because deleting tests looks like maintenance.

It is not maintenance. Every eval case is a guarantee the team made to itself: this failure mode will not recur silently. Deleting the case retires the guarantee. The pass rate does not change, the dashboard stays green, and the only thing that disappears is the team's memory that the guarantee ever existed. Six months later a model migration reintroduces exactly the regression a deleted case was guarding, the postmortem rediscovers a lesson the team already paid for once, and someone writes "we should add a test for this" — the test that was deleted in the cleanup PR.

The eval set most teams run as their regression suite was hand-curated by an engineer in week three of the project, frozen by week six because nobody wanted to touch it before launch, and is now being used in month nine to gate deploys. The product has shifted twice. The user base has tripled. The cases the LLM actually sees in production overlap with that frozen suite by maybe forty percent. When the suite passes, nobody trusts it; when it fails, nobody knows whether the failure is real or whether the case is just stale. The team writes a doc proposing a "v2 eval set" and never gets around to it.

Meanwhile, every request the system has handled in production has been recorded in a tracing backend. Every prompt, every tool call, every intermediate output, every refusal, every retry — all of it sitting in object storage, time-indexed and span-tagged, ready to be replayed. The highest-fidelity test corpus the team will ever have is already on disk. They built an eval suite from scratch instead of reading from it.