383 posts tagged with "ai-agents"

Your eval dashboard is green. Turn-level accuracy sits at 95%, the LLM judge agrees with your annotators, and every regression test passes on the way to production. Then a user files a bug: the agent recommended a Postgres index on turn nine that directly contradicted the "we're on DynamoDB" constraint the user set on turn one. You pull the transcript. Every single turn, read in isolation, is a reasonable response. The conversation, read as a whole, is a disaster.

This is the central lie of turn-level evaluation. It grades request-response pairs because that's the unit that's cheap to label, and it quietly assumes that a conversation is just a bag of independent turns you can average. It isn't. The response at turn nine is conditioned on everything that came before it, and the failures that actually reach users almost never live inside a single turn — they live in the seams between turns, where state gets dropped, assumptions harden, and small errors compound into a wrong final answer.

You instrumented your agent the way every observability guide told you to. Every turn gets a span: the full prompt, the retrieved chunks, the model's chain of thought, the tool calls it considered, the plan it drafted and then abandoned, the final action. Storage is three cents a gigabyte and the traces have saved you a dozen times during incident response. You kept everything because keeping everything was free and occasionally heroic.

Here is the part nobody on the platform team modeled: that trace is written in fluent English, it speculates about your customers by name, and it drafts actions you never actually took. It is the single most quotable artifact your company produces, and a plaintiff's attorney will read the juiciest 200 words of it to a jury with your logo on the slide behind them. The reasoning trace that makes your system debuggable is the same reasoning trace that makes it liable.

"Be conservative in what you send, be liberal in what you accept." Postel's law is arguably the most successful design principle in the history of networking — it's how TCP implementations from different vendors managed to interoperate in the 1980s, and it shaped four decades of protocol and API design. It is also, in the judgment of the IETF itself, a principle that curdles over time: RFC 9413 grew out of a draft bluntly titled "The Harmful Consequences of the Robustness Principle," arguing that liberal acceptance helps interoperability in the short term while quietly rotting the ecosystem in the long term.

Agent systems compress that long-term rot into weeks. When the "sender" is a language model emitting tool calls, every act of liberal acceptance — casting "5" to 5, dropping an unknown field, fuzzy-matching an enum typo — destroys the exact signal you need to keep the system healthy. The tool boundary is the one place in an agent architecture where failing loudly is the reliability feature, and most teams get it backwards.

Your CODEOWNERS file encodes an assumption so old nobody remembers making it: the author of a diff knows whose turf they're on. A human engineer changing the billing service knows they're in billing territory. They scoped the change to their team's directories on purpose, and if they had to touch another team's code, they walked over — physically or on Slack — and gave a heads-up before the review request ever fired. Path-based review routing works because human authors pre-route their own changes.

Agents don't do this. An agent told to "migrate every deprecated logger call" sweeps the whole repository in an afternoon and produces a diff touching four hundred directories. GitHub dutifully consults CODEOWNERS, matches every glob, and fans the review request out to forty teams simultaneously.

Nobody asked those teams whether they wanted this change. Nobody warned them it was coming. And the "author" requesting their approval isn't a colleague they can grab for context — it's a bot account acting on behalf of someone in a different org who may not even know which teams got paged.

You ran the deprecation playbook flawlessly. Announcement email six months out. A migration guide with code samples in four languages. Sunset headers on every v1 response. A banner in the developer dashboard. Two reminder emails. Then you turned off v1 — and instead of the quiet cutover the playbook promised, your error rate went vertical and stayed there. Traffic didn't migrate. It just kept arriving, failing, and retrying, at higher volume than before, because every failure triggered another attempt.

The playbook didn't fail because you executed it badly. It failed because it assumes a human is on the other end — someone who reads email, skims changelogs, and files a Jira ticket to migrate before the deadline. A growing share of your traffic has no such person behind it. It comes from agents whose knowledge of your API is frozen in system prompts, tool schemas, and model training data scattered across thousands of repositories you can't see and will never reach. Your sunset notice was written for a reader. Your biggest client is a prompt.

Your agent's internal metrics look great. Task completion is at 94 percent. The #agent-feedback Slack channel has been quiet for three weeks. Leadership is ready to turn it loose on customers. Then external users arrive, and within a month the numbers collapse: escalations spike, trust craters, and the postmortem question everyone asks is "how did dogfooding miss this?"

Dogfooding didn't miss it. Dogfooding hid it. Internal users are not a miniature version of your customer base — they are a population of expert operators who silently repair the agent's mistakes, learn which prompts to avoid, and share workarounds in DMs instead of filing bugs. Every repair makes the dashboard look better and the product worse-understood. The clean signal you're reading isn't quality. It's compensation.

A human engineer who sees a test fail on a change that couldn't possibly have caused it does something an agent cannot: they shrug. They hit rerun, mutter about the CI gods, and move on. That shrug encodes years of accumulated context — this test has been flaky since March, that service's staging environment falls over on Mondays, nobody trusts the WebSocket suite. A coding agent has none of this. It sees a red X and takes it as ground truth, because everything in its training and its prompt tells it that a failing test means the code is wrong.

What happens next is the expensive part. The agent doesn't shrug — it acts. It "fixes" code that was never broken. It reverts a correct change because the suite went red after applying it. It burns through its token budget chasing a phantom, adding retries and sleeps and defensive checks to code paths that were fine, until the flaky test happens to pass and the agent concludes its last mutation was the cure. Nondeterminism in your test substrate was always a tax on human attention. For agent loops, it's something worse: corrupted training signal, injected directly into the decision-making of a system that acts on it at machine speed.

Classic chaos engineering rests on a quiet assumption: if you inject the same fault twice, you get the same failure twice. Kill the pod, watch the failover, fix the gap, kill the pod again to confirm. The entire discipline — hypothesis, blast radius, steady-state metrics — presumes a system deterministic enough that the experiment is repeatable.

Agent systems break that assumption at the root. Inject a tool timeout into an agent run and the model reroutes its plan — one time it retries, another time it substitutes a different tool, a third time it confidently fabricates the result it never fetched. Run the identical fault against the identical prompt and you get a different trajectory, because the failure path runs through a stochastic planner. The failure you saw in production last Tuesday will never happen again in exactly that shape. And that's precisely why you have to rehearse it anyway.

GraphQL's core bet was that clients should compose their own data requirements. For a decade, that bet mostly lost — because the clients were human teams who didn't want to compose anything. Frontend engineers wanted a stable endpoint they could call and forget. The flexibility GraphQL sold was a tax they paid in resolver complexity, caching workarounds, and security review, in exchange for a benefit — per-request field selection — that a known, slow-changing web app barely needed. By 2024, the practitioner consensus had visibly cooled: most internal APIs serve two or three known clients, and a well-shaped REST endpoint or a BFF layer covers them fine.

Then a new kind of client showed up. An AI agent doesn't have a fixed set of screens. It decides, per task, what data it needs, and it pays for every byte of the response — literally, in tokens, and cognitively, in degraded reasoning as the context window fills with fields nobody asked for. The client that actually composes its own data requirements finally exists. It just isn't human.

The first fifteen minutes of almost every incident are mechanical. Pull the four graphs everyone always pulls. Diff the deploys against the incident start time. Check which feature flags flipped today. Search the runbook wiki for the error string. None of this requires judgment — it requires being awake, and at 3 AM your on-call engineer is spending those minutes finding their laptop, joining the bridge, and remembering which dashboard is the real one. An agent can have all of it done before the human's screen unlocks.

Yet in most organizations, the conversation about AI in incident response dies at a single anecdote: "we heard about a team whose auto-remediation script took down production." So the whole idea gets banned — not scoped, not staged, banned. This is a category error. The horror stories are about the top rung of an autonomy ladder, and teams respond by refusing to step on the bottom rung, where the agent has no write access to anything and the worst it can do is post a wrong paragraph in Slack.

Every no-code platform you've ever used was priced against the same benchmark: the cost of hiring an engineer. That was the entire pitch. Writing real software required scarce, expensive people, so the platforms sold you a trade — give up flexibility, accept the walls of the sandbox, and in exchange your ops manager can ship the inventory dashboard herself instead of waiting six months in the engineering backlog. The trade made sense for fifteen years because the benchmark held. Code stayed expensive.

Then the benchmark collapsed. When a coding agent can build and maintain a real application — actual code, in a repository, with a schema you own — for less than the monthly cost of a Retool seat, the drag-and-drop abstraction stops being a shortcut and starts being a ceiling. You're now paying a premium to be prevented from having real software.

But here's the part most obituaries get wrong: the platforms won't die. The smart ones are already pivoting to the thing that was never about the cost of code — governed data access, blessed integrations, and being the sandbox that IT already approved. The constraint on internal tools is moving from "who can build it" to "who's allowed to," and that shift rewrites how you should evaluate your entire internal-tools stack.

Give a coding agent a Python module and it operates on solid ground: the file on disk is the program. Read it, edit it, run it, observe the result — the loop closes. Give the same agent a Jupyter notebook and every one of those assumptions quietly breaks. The agent edits cell 12 with full confidence, not knowing that you re-ran cell 3 an hour ago with different data, that a variable defined in a since-deleted cell is still alive in the kernel, and that the output it just read under cell 7 was produced three kernel restarts ago by code that no longer exists.

A notebook is a REPL wearing a file costume. The .ipynb on disk looks like source code, but the thing that actually determines behavior — the kernel's accumulated memory — is invisible, unserialized, and shaped by the exact sequence of human clicks that produced it. Agents are trained on the contract that code determines behavior. Notebooks void that contract, and most agent harnesses don't even know it.