68 posts tagged with "testing"

Your eval dashboard is green. Turn-level accuracy sits at 95%, the LLM judge agrees with your annotators, and every regression test passes on the way to production. Then a user files a bug: the agent recommended a Postgres index on turn nine that directly contradicted the "we're on DynamoDB" constraint the user set on turn one. You pull the transcript. Every single turn, read in isolation, is a reasonable response. The conversation, read as a whole, is a disaster.

This is the central lie of turn-level evaluation. It grades request-response pairs because that's the unit that's cheap to label, and it quietly assumes that a conversation is just a bag of independent turns you can average. It isn't. The response at turn nine is conditioned on everything that came before it, and the failures that actually reach users almost never live inside a single turn — they live in the seams between turns, where state gets dropped, assumptions harden, and small errors compound into a wrong final answer.

You can stand up a staging database. You can stand up a staging queue, a staging payments sandbox, a staging copy of every third-party API you depend on. For thirty years the whole discipline of pre-production has been built on one assumption: that you can create a faithful-enough replica of production, poke it, and learn something true about what will happen when you ship.

Then you added a hosted model to your critical path, and the assumption quietly broke. The one component whose behavior now dominates your product — the thing that decides what your app actually says and does — is the one component you cannot stand up a staging copy of. It is versioned by someone else, rate-limited by someone else, and quietly updated by someone else on a schedule you don't see. Your staging environment has a staging everything, except a staging model.

A human engineer who sees a test fail on a change that couldn't possibly have caused it does something an agent cannot: they shrug. They hit rerun, mutter about the CI gods, and move on. That shrug encodes years of accumulated context — this test has been flaky since March, that service's staging environment falls over on Mondays, nobody trusts the WebSocket suite. A coding agent has none of this. It sees a red X and takes it as ground truth, because everything in its training and its prompt tells it that a failing test means the code is wrong.

What happens next is the expensive part. The agent doesn't shrug — it acts. It "fixes" code that was never broken. It reverts a correct change because the suite went red after applying it. It burns through its token budget chasing a phantom, adding retries and sleeps and defensive checks to code paths that were fine, until the flaky test happens to pass and the agent concludes its last mutation was the cure. Nondeterminism in your test substrate was always a tax on human attention. For agent loops, it's something worse: corrupted training signal, injected directly into the decision-making of a system that acts on it at machine speed.

Your own stack has three environments. Dev is disposable, staging is production-shaped, and prod is sacred. Twenty years of engineering culture — CI gates, canary deploys, blue-green rollouts — all rest on that tiering. Then you give an agent a tool that calls Salesforce, QuickBooks, or Gmail, and the tiering silently evaporates. There is no staging Salesforce for your customer's org. There is no shadow copy of the invoice ledger. The moment a tool call crosses your network boundary into a third-party SaaS, there is exactly one environment, and it is production.

This is the least discussed gap in agent engineering. We have gotten good at sandboxing the agent's compute — containers, egress allowlists, resource caps. We have gotten passable at evaluating the agent's reasoning — offline evals, LLM-as-judge, trajectory scoring. But the agent's actions on the outside world still run against live systems holding real customer data, because for most SaaS surfaces nothing else exists. Teams quietly resolve this the only two ways they can: test against production, or don't test at all.

The CI test was a single assertion: same model, same temperature, same prompt, same seed, same output string. It passed on every developer's laptop, passed on the first hundred CI runs, and then flaked once every fifty runs for three weeks before anyone admitted the pattern was real. The first hypothesis was the obvious one — a non-deterministic dependency somewhere in the test harness — and three days of investigation found nothing. The actual cause was sitting in a footnote on the provider's API reference: "seed provides best-effort determinism." The team had read the parameter name and assumed a contract. The provider had documented a hint.

This is a specific failure mode of hosted inference that catches teams who design test infrastructure around a single mental model: the model is a pure function of its inputs, and the seed is what makes the function reproducible. Both halves of that model are wrong in production, and the gap between the API surface and the underlying physics is wide enough that teams build entire eval and regression-test stacks on top of an assumption their provider explicitly disclaimed.

The agent demo always works. Three customers in the table, one matching record, twelve documents in the vector index, an empty calendar with infinite open slots. The agent picks the right row, retrieves the right document, books the right meeting. Ship it.

Then production hands the same agent ten million customers with three "John Smith"s in the same city, a filter that returns four thousand rows because the agent confidently wrote status != 'closed' when it meant status = 'active', a vector query that returns seven plausible documents the agent has never had to choose between, and a calendar where every slot is a negotiation. The capability that looked correct in dev is qualitatively different in prod — not slightly worse, not flakier, but solving a different problem the dev environment never made it solve.

This is the gap that "it worked locally" hides. For deterministic code, that phrase is already a lie about edge cases. For agents, it is a stronger lie, because the agent's behavior is a function of input distribution, and the input distribution shifts from "trivial" to "ambiguous" the moment you cross the prod boundary.

Open the resilience suite for your agent and look at what it actually tests. You will find timeouts. You will find connection drops, 500s, rate-limit responses, malformed JSON, maybe a tool that hangs for thirty seconds before failing. All of it is fault injection in the classic mold: the tool is broken, and the question is whether your agent degrades gracefully.

Now look for the test where the tool is not broken at all. The one where the tool responds in 80 milliseconds, returns perfectly valid JSON against the schema, and the value inside is simply wrong. A balance that is stale by three days. A customer record with two fields swapped. An order quantity with two digits transposed. An empty result list for a query that should have returned forty rows.

You will not find it. Almost nobody injects that fault. And it is the one fault your agent is least equipped to survive, because every other fault announces itself and this one does not.

You wrote the eval set eighteen months ago. Back then it was a useful instrument: the cheap model scored 71%, the better one scored 84%, and when a regression slipped in the number dropped and somebody noticed. The suite earned its place in CI. You stopped thinking about it.

Run it today and every candidate model scores 96, 97, 98. The new release scores the same as the old one. The model you suspect is worse scores the same as the model you suspect is better. The number still renders green in the dashboard, the check still passes, and it tells you exactly nothing. Your eval set didn't break. It got easier — because the models got better underneath it — and nobody was watching the moment it stopped discriminating.

This is eval saturation, and it is not a failure mode you might hit. It is the guaranteed end state of any static suite given a long enough timeline. A test that everything passes has stopped being a test.

Every mature AI team builds its eval suite the same way, and almost nobody says the quiet part out loud. A failure shows up in production. Someone writes a postmortem. An engineer distills the incident into a test case, adds it to the eval suite, and the dashboard goes green again. Repeat this loop for a year and you have a few hundred cases, a satisfying pass rate, and a deeply comforting number to put on a slide.

Here is the quiet part: that suite is a museum. Every exhibit is a failure class the team has already survived. A 98% pass rate certifies your system against the past — against the specific ways it has already broken — and says almost nothing about the novel failure mode that a model migration, a prompt edit, or a shift in user behavior is about to introduce. The eval set is a lagging indicator wearing the costume of a leading one.

Look at where most agent eval sets come from. Someone builds the agent, demos it to the team, the demo works, and the demo script becomes the eval suite. The cases that pass review are the cases someone already watched pass. The eval set is, almost by construction, a recording of the happy path — the one tool sequence that worked the day the screenshot was taken.

So when the dashboard says the agent scores 94%, what it actually says is: it passes the cases we imagined. It says nothing about the case where the search API returns a 429 in the middle of a multi-step plan, where the user contradicts a constraint they stated two turns ago, or where retrieval comes back empty and the agent has to decide between guessing and admitting it doesn't know. Those cases aren't failing your eval. They were never in it.

This is golden-path bias, and it is the default shape of an agent eval suite unless you fight it deliberately. The fix is not more cases. It is different cases — chosen by failure mode, harvested from production, and stress-tested with deliberate faults.

Ask a coding agent to "add tests for this module" and you will get tests. They will be neatly formatted, they will follow your project's conventions, and they will pass. Coverage will tick up. The PR will look like diligence. And a meaningful fraction of those tests will not be able to catch a single bug you might plausibly introduce.

This is not a story about a model being dumb. The agent did exactly what it was asked. The problem is that "add tests" and "add tests that constrain the behavior" are different requests, and only one of them is easy to verify at a glance. A green checkmark looks identical whether it sits on top of a real assertion or a tautology.

The result is a suite that grows in line count and shrinks in power. You end up with more files, more CI minutes, more things to maintain — and roughly the same probability of shipping a regression as before you started.

Your eval suite passed. All 240 cases green, same as last week. You ship. Two days later support tickets spike, and when you read the transcripts you find a failure mode your suite has no opinion about at all — not a case that flipped from pass to fail, but a question your users started asking that your suite never thought to ask.

This is the quiet failure of evals. We treat a green run as a statement about the present: "the system works." It is actually a statement about a past — the moment the eval cases were written. An eval authored six months ago encodes three things as they were that day: the product's scope, the model's failure modes, and the way real users phrase their requests. All three move. The feature grew a new surface. The model got upgraded twice. The input distribution drifted as users learned what the product could do. The suite did not move with them, so a green run increasingly certifies a world that no longer exists.

Nobody notices, because nothing breaks. A stale eval does not throw an error. It keeps passing, confidently, while measuring less and less of what matters.