The Meeting That Should Have Been an Eval

Every team shipping an LLM feature eventually holds the same meeting. Someone asks whether the new model — or the new prompt, or the new retrieval tweak — is good enough to ship. The senior engineer who spent the weekend testing says it feels sharper. The PM says a customer complained about exactly this last week. The skeptic on the team pulls up a transcript where the old version was clearly better. Forty-five minutes later, nobody has changed their mind, and the decision gets made by whoever talks last or outranks the room.

That meeting is a symptom. It recurs because the team is trying to settle an empirical question — did this change make the system better or worse? — with anecdotes, and anecdotes don't converge. You can stack them all day. The reason the debate never ends is that there is no shared instrument that everyone agrees to be bound by. The meeting that should have been an eval is the meeting where you discover you're missing one.

Why Anecdotes Can't Settle the Question

Human intuition about model quality is real signal, but it's noisy and biased in predictable ways. The engineer who tested over the weekend ran maybe thirty queries, all of which they chose, most of which exercise the cases they personally care about. The PM remembers the failure that generated a Slack thread, not the hundred silent successes around it. The skeptic found one regression and, having found it, is now anchored on it.

None of these people are wrong. They're each holding a different, small, non-random sample of the output distribution, and they're arguing as if their sample were the population. That's why the meeting doesn't converge: you can't average three biased samples by talking louder. You'd need to compare the same inputs across both versions and count, which is exactly the thing nobody did before the meeting.

There's a deeper problem. "Good enough to ship" is not one question. It's bundling at least three: did average quality go up, did any specific behavior we care about regress, and did the cost or latency move in a direction we can't afford. A model upgrade can lift average quality while quietly breaking your most sensitive workflow. A widely-cited example: a team about to swap a model for a newer, cheaper version that topped the public leaderboard ran their own task suite and found the new model was three points worse on their actual workload — and emitted forty percent more insecure code patterns on their API-handling tasks. The leaderboard said upgrade. Their eval said don't. Without the eval, that swap ships on vibes and the regression surfaces in production two weeks later.

What the Meeting Is Really Asking For

Strip away the personalities and the recurring debate is asking for one thing: a number that everyone agrees, in advance, to treat as the answer. Not a perfect number — a shared one. The whole value of an eval is that it converts "the agent feels worse" into something every person in the room is bound by, because they signed off on the metric before they saw the score.

This is the same move software made decades ago with regression tests. Nobody convenes a meeting to debate whether a refactor broke the login flow. There's a test. It's green or it's red. The argument was settled the day someone wrote the test, and every subsequent change inherits that settlement for free. LLM features feel exempt from this because their outputs are fuzzy and subjective, so teams conclude they can't be tested and fall back to meetings. That conclusion is wrong. Fuzzy outputs are harder to grade, not impossible, and "harder" is an engineering problem with known solutions.

The reframe that unlocks everything: the recurring shipping debate is not a decision to be made well. It's a metric that hasn't been built yet. Every time the meeting happens, it's generating the exact dataset you need — real disagreements about real inputs where reasonable people reached different verdicts. Those are your hardest eval cases, handed to you for free. The meeting is the data collection step of an eval you keep refusing to write down.

Turning the Argument Into a Standing Metric

Loading…