151 posts tagged with "evals"

A support agent is about to issue a refund. Before it fires the tool call, your team added a gate: only proceed if the model says it's at least 90% confident. The model dutifully returns "confidence: 0.95," the refund goes out, and the citation it used to justify the amount — a policy clause about damaged goods — does not exist. It never did. The model invented the clause and then rated its own invention at 95%.

This is the trap. Teams reach for the model's confidence number because it looks like the probability you'd get from a calibrated classifier — a value where 0.9 means "right nine times out of ten." It is not that. An LLM's self-reported confidence is a fluent-sounding token like any other, shaped by tone, phrasing, and training incentives that have almost nothing to do with whether the underlying claim is true.

If you are gating real actions on that number, you are gating on a vibe.

Your agent's internal metrics look great. Task completion is at 94 percent. The #agent-feedback Slack channel has been quiet for three weeks. Leadership is ready to turn it loose on customers. Then external users arrive, and within a month the numbers collapse: escalations spike, trust craters, and the postmortem question everyone asks is "how did dogfooding miss this?"

Dogfooding didn't miss it. Dogfooding hid it. Internal users are not a miniature version of your customer base — they are a population of expert operators who silently repair the agent's mistakes, learn which prompts to avoid, and share workarounds in DMs instead of filing bugs. Every repair makes the dashboard look better and the product worse-understood. The clean signal you're reading isn't quality. It's compensation.

Somewhere in your codebase there is a config file that says something like model: small-and-cheap under the test profile and model: frontier under production. It felt responsible when someone added it — why burn frontier-model tokens on CI runs that fire twenty times a day? But that one line quietly repealed a rule your team has followed for fifteen years without thinking about it: the environment you test in should behave like the environment you ship to.

The twelve-factor methodology called this dev/prod parity, and we got so good at it that we stopped noticing it. Docker gave us bit-identical runtimes. Infrastructure-as-code gave us identical topology. Then we put a language model in the middle of the request path and reintroduced the exact gap we spent a decade closing — except this time the divergent component isn't a database version. It's the part of the system that makes the decisions.

Your pricing page says every user pays the same. Your cost dashboard says otherwise. The same AI feature — same prompt template, same model, same feature flag — costs 55% more to serve a Spanish user, roughly double for a Japanese user, and over 3x for an Arabic or Bengali user. Meanwhile, the quality those users receive is measurably worse: on identical benchmark questions translated across languages, frontier models drop 13 to 24 percentage points when you leave the English distribution.

Most teams shipping AI features globally have never measured either number. They have per-locale pricing, per-locale support SLAs, per-locale legal review — and a single English eval suite standing in for the experience of every user on the planet.

This is the tokenizer tax, and it compounds with a quality gap that scale alone doesn't close. Both are invisible in your dashboards until you slice by language, and both were decided years before you wrote your first prompt, by the training corpus of a tokenizer you don't control.

Six weeks after launch, your quality dashboard starts sagging. Thumbs-down rates creep up, task completion drifts down, and the on-call channel fills with screenshots of bad responses. The team does what teams do: they diff the prompts (unchanged), check the model version (pinned), audit the retrieval index (fresh), and bisect the deploy history (nothing shipped). Everyone concludes the model provider silently degraded the model. The provider, of course, insists nothing changed.

Everyone is looking in the wrong place. Nothing in the system changed. The users did.

Launch-week metrics assume launch-week users. But people adapt to an AI product within weeks, and they adapt in ways that systematically break the assumptions baked into your prompts, your evals, and your launch benchmarks. Your model is frozen. Your users are not. The gap between them is a form of drift that most teams don't instrument for at all — and it produces the most confusing incident pattern in AI engineering: metric decay with no deploy.

Your eval suite reports a number. Behind that number is a label. Behind that label is a person — usually one you have never met, working a two-week gig managed over WhatsApp, paid through a mobile money app, ranking model outputs they were given fifteen seconds to read. The eval score you ship to your VP, the regression gate that blocks your deploy, the leaderboard rank you put in the launch blog — all of it inherits the quality of that person's attention on that afternoon.

We talk about evals as if they were instruments: calibrated, repeatable, objective. They are not. An eval is a measurement device whose sensor is a human labor pipeline, and most teams budget for the harness while treating the people as a free input. That accounting error is why eval scores drift, why your "ground truth" disagrees with itself, and why the most expensive bottleneck in frontier AI is no longer compute.

A VP of Engineering decides the company needs to "do AI." There is already a machine learning team — three people who built the recommendation model, tuned the fraud classifier, and maintain a feature store. The obvious move is to point them at the new LLM initiative. They know the math. They've shipped models. How different could it be?

Six months later the prototype demos beautifully and dies in production. Nobody can say why the agent occasionally books the wrong meeting, the cost per request is four times the estimate, and there is no way to tell whether last week's prompt change made things better or worse. The ML team is frustrated because none of the tools they're good at — gradient descent, data pipelines, hyperparameter sweeps — apply to a model they can't retrain and can't see inside.

This is the most common org mistake in AI right now, and it comes from a reasonable-sounding assumption: that the craft of building models and the craft of building on models are the same job with a different label. They are not. They overlap less than "frontend engineer" and "backend engineer" do.

The demo worked. You typed a question, the agent called three tools, reasoned through a multi-step plan, and produced an answer that made the room lean forward. Someone said "ship it." Three months later you're still not shipped, and nobody can quite explain where the time went.

Here is where it went: the demo was 10% of the work. It was the brain. The other 90% is the plumbing — the evals, guardrails, observability, cost controls, and fallback paths that nobody claps for because they're invisible when they work and catastrophic when they don't. That 90% is the demo-to-production tax, and most teams budget for it as if it were a rounding error.

The numbers say otherwise. MIT's 2025 study of enterprise AI found that 95% of generative-AI pilots delivered no measurable P&L impact. A separate analysis put it more bluntly: for every 33 proofs of concept an enterprise starts, four reach production. That's an 88% mortality rate, and the cause of death is almost never the model. It's everything the demo let you skip.

The chunker started prepending [line N] to every chunk. The eval went green. Every citation the model produced after that day pointed to the paragraph one position before the actual evidence, on every document, in the regulated industry the product serves. The team did not find out from the eval. The team found out from an auditor who looked at the cited sentence, read it, and pointed out that it contradicted the claim it was supposed to support.

This is the kind of regression that survives a code review, a manual QA pass on three sample documents, and a feature-flag rollout. None of those checks were wrong in isolation. They were all asking the same question — does a citation appear where one is expected — and none of them were asking the question the auditor asked, which is whether the citation points at the sentence the claim came from. The gap between those two questions is where the off-by-one lived for as long as it lived.

What makes this failure mode worth a separate write-up is not the bug itself. Off-by-one errors are old news. The interesting part is that the failure was produced by two systems that continued to agree on the structure of an integer while silently disagreeing about what the integer meant.

The eval dashboard had been climbing for three sprints. Quality up six points on the hard slice, up nine on the regression slice, up twelve on the slice the support team had hand-curated from last quarter's worst tickets. The team shipped a model promotion off the back of it. Two days later, a customer asked a question that looked nothing like anything in the eval set, and the answer was worse than what they had been getting six months ago.

The forensic was quick once someone thought to run it. The prompt engineers had been working out of the same repo as the eval team. They had found the curated examples — the painstaking ones, the ones where someone had argued for an hour about the correct phrasing of the ideal answer — and over a few sprints they had copy-pasted the strongest of them as few-shot demonstrations into the production system prompt. The dashboard kept going up because the model was being graded on inputs it had seen verbatim at inference time. Nobody flagged it. Nobody owned the boundary between "the examples we measure quality against" and "the examples we ship in the prompt." Both teams were doing exactly the job they had been hired to do.

Your reliability dashboard says 99.95%. Your support inbox says something else. Twice a week, for ten or twenty minutes at a time, a thin sliver of users gets a version of your product that talks like a different company. The refusals read funny. A structured field that always rendered as a tidy two-column card now shows up as a paragraph with bullet points smashed inside it. Tone shifts from "calm expert" to "eager assistant." Nobody opens a ticket — they just close the tab and try again later.

Your provider went down. The failover worked. Latency stayed under SLO. The error budget did not move. And the experience your users got during that window was not the one you ship.

The mental model most teams carry into multi-provider architecture is that the system prompt is portable — a contract negotiated with the abstract idea of "a capable model," readable by anyone who speaks the LLM dialect. That model is wrong. A system prompt is a tuned artifact. It is tuned against a specific model's preferences, refusal grammar, formatting habits, and instruction-following biases. When the failover engages, you are not handing the same contract to a comparable counterparty. You are handing a contract written in your primary's idiom to a model that reads a different idiom and signs it anyway.

A user submits a question. Your model produces an answer that is confidently wrong in a very specific way: the format is perfect, the reasoning is well-structured, and a particular qualifier — one that does not apply to this question at all — appears in exactly the place a similar qualifier appeared in example three of your system prompt. Not a hallucination. Not a prompt injection. The model did precisely what the examples taught it to do, on a question those examples were never meant to cover.

This is the failure mode that few-shot prompting actively encourages and that most eval suites are structurally blind to. Your examples are not neutral demonstrations of "what good looks like." They are case law. The model selects the closest match by surface tokens and applies the precedent — including its constraints — to whatever case is in front of it.