278 posts tagged with "reliability"

A researcher agent asks a retrieval agent for a document. The retrieval agent, mid-task, decides it needs the researcher to clarify the query before it can search. The researcher, waiting on the document, won't respond until it has the document. Neither one is broken. Neither one is looping. They are both politely, indefinitely, waiting for each other — and your orchestrator, which has no concept of "both of these are blocked on each other," will happily hold that state until a timeout you never configured finally fires, or until a human notices the run has been "in progress" for forty minutes.

This is a deadlock. It is one of the oldest failure modes in computing, and it has nothing to do with how smart your model is. It is a property of how work is coordinated, not how work is done. The uncomfortable finding from the last year of multi-agent research is that most of what breaks in agent swarms breaks here, in the coordination layer, not in the reasoning of any single agent.

Single-agent thinking never surfaces these bugs. When one model runs a loop of tool calls, the worst it does is spin — and a spinning loop is at least visibly spinning. The moment you have two or more agents that can wait on each other, you have inherited the entire back catalog of distributed-systems pathologies: circular wait, livelock, lost messages, premature termination, races on shared state. Nobody sat down and decided to build a distributed system. You built one anyway the day you added a second agent.

When your agent issues a refund, sends an email, closes a ticket, or writes a row, that action leaves the system and enters the world. The world does not have a rollback button. The customer already saw the refund. The recipient already read the email. And when the agent takes a wrong turn three steps later, your recovery plan is usually a sentence in a retro: "we told it not to do that again."

"Don't do that again" is not undo. It is a promise about the future applied to a problem in the past. The uncomfortable truth is that most agent stacks have no mechanism to reverse a completed side effect — not a bad mechanism, no mechanism. The agent can plan, call tools, and retry, but it cannot walk backward. It has a forward gear and no reverse.

When you deprecate an API for human developers, you have a whole ceremony for it. You bump the version, add deprecated: true to the OpenAPI spec, ship a Sunset HTTP header, send an email to the developer mailing list, post to the changelog, and give people six months to migrate. The signal reaches a human who reads it, files a ticket, and updates their client before the old path disappears.

Now point that same deprecation at an agent. The model calling your tool does not read your changelog. It does not subscribe to your mailing list. It never sees the Sunset header unless you deliberately put it somewhere the model looks, and even then it has no reliable habit of acting on it. The deprecation notice you so carefully authored lands in a mailbox with no reader. The agent keeps calling the old shape of the tool until the shape is gone, and then it fails — often silently, often in production, often at 2 a.m.

This is the quiet asymmetry of building tools for agents instead of people. Every discipline we built over two decades of API evolution assumes a human sits between the deprecation and the migration. Take the human out, and the entire mechanism goes dark.

Every agent architecture diagram has the same three boxes. There's the happy path, where the model answers and the user leaves satisfied. There's the automatic fallback, where a low-confidence answer triggers a retry, a different tool, or a canned "let me look into that." And there's a third box, usually drawn last and smallest, labeled escalate to human. Everyone nods at that box in the design review. It looks like closure — the safety valve that makes the whole system defensible. "Don't worry, if the agent can't handle it, a person takes over."

Then you ship, and you discover the box was a lie. Not a technical lie — the code works, the ticket gets created, the conversation gets flagged. A staffing lie. The arrow labeled escalate to human points at a queue that nobody owns, has no service-level agreement, and appears on no one's on-call rotation. The agent did exactly what it was told. It handed the problem to an organization that never agreed to catch it.

The most expensive bug in your agent isn't a hallucination. It's a retry.

Somewhere in your stack there is a tool that charges a card, sends an email, closes a ticket, or writes a row. The agent calls it, the call takes too long, a timeout fires, and the agent — being a good, resilient piece of software — calls it again. The catch is that the first call already succeeded. The response just never made it back. Now you've charged the customer twice, and no amount of prompting "please be careful with payments" was ever going to prevent it.

This is the oldest failure mode in distributed systems wearing a new outfit. We solved it for HTTP APIs a decade ago with idempotency keys. But most agent stacks reintroduced the problem by pointing retry logic built for reads at tools that do writes, and then never sent the one field that would have made the retry safe.

The demo worked. The beta worked. Then the launch drove ten times the traffic into a token-per-minute quota you never load-tested against, and every user past the ceiling got a 429 at the exact moment you had the most eyes on the product.

This is the failure mode nobody rehearses. Teams load-test their own servers obsessively — replicas, connection pools, database indexes — and then route every request through a provider quota that lives in someone else's account, at a ceiling they've never actually touched. The rate limit isn't an error to catch in a try block. It's a hard product constraint, and if you haven't planned around it, the launch is the first time you'll find out where it sits.

A support request comes in: "My subscription got charged twice, cancel one and refund it." A router agent classifies it as billing and hands off. A billing specialist looks up the account, confirms two charges, and hands the refund to a tool-calling sub-agent. The sub-agent issues one refund, then a summarizer writes back to the customer: "Done — your duplicate charge has been refunded." Every step is individually correct. The customer was charged three times, not two. One duplicate is still on their card, and the summarizer had no way to know, because by the time the conversation reached it, the number "2" had already hardened into fact four steps upstream.

Now run the postmortem. Whose bug is it? The router classified correctly. The specialist read the two charges it was given. The tool agent refunded exactly what it was told. The summarizer summarized accurately. Pull up each agent's trace in isolation and every one of them passes. The system failed and no component did. This is the conversation with no owner, and it is the defining operational failure of multi-agent architectures — not a model being dumb, but responsibility diffusing across a handoff chain until no single agent is on the hook for the outcome the user actually experienced.

The incident starts the way most of these do: a customer-facing feature throwing 429s in production at 2 p.m. on a Tuesday, no deploy, no traffic spike, nothing in the feature's own logs to explain it. The on-call engineer stares at the dashboard for twenty minutes before someone in another channel mentions, almost in passing, that they kicked off a "quick backfill" to re-summarize a few hundred thousand old documents. Two teams, two codebases, two on-call rotations — and one API key's worth of rate limit between them. The backfill ate the budget. The chatbot starved.

This is the noisy neighbor problem, and the twist that makes it so dangerous with model APIs is that the neighbor isn't some anonymous tenant on shared cloud hardware. The neighbor is another team at your own company, and the wall between you is thinner than anyone thinks.

The incident bridge is quiet in a way that means everyone is stuck. A support ticket says the agent told a customer their refund was approved when it wasn't. You have the full trace: the prompt, the retrieved account record, the tool calls, the model's reasoning, the final message. You replay it. The agent does the right thing. You replay it again. Right again. Nine times out of ten, the trace that produced the incident produces a correct answer instead. Someone on the call finally asks the question that the retro template can't handle: so what was the root cause?

There isn't one. Not in the sense the template means. The five-whys chain runs "the agent told the customer the wrong thing" → "because the model generated an approval" → "because it sampled a token sequence that asserted approval" → "because... that's what the probability distribution allowed." The last why bottoms out in a shrug. "The model sampled a bad token" is technically true and operationally useless. It names no fix, assigns no owner, and closes no gap. You can write it in the report, but everyone reading it knows you've documented a coincidence, not a cause.

You wired up a second model vendor because the first one went down. It was a Tuesday, the primary provider's status page was a wall of red for a few hours, and your feature was dead in the water with it. So you did the responsible thing: you added a fallback. Now, if OpenAI is unreachable, you route to Anthropic. If Anthropic rate-limits you, you fall back to Gemini. The architecture diagram looks clean and grown-up. Reliability, solved.

Except you didn't add a replica. You added a second opinion. And a second opinion is a very different thing to operate than a second copy.

The mental model you imported comes from the stateless-service playbook: run three identical instances behind a load balancer, and if one dies the other two serve the exact same responses. That works because the replicas are byte-for-byte interchangeable. Two language models from two vendors are not. They were trained on different data, tuned with different objectives, and they disagree — systematically, not randomly — on exactly the inputs where your users notice.

"Be conservative in what you send, be liberal in what you accept." Postel's law is arguably the most successful design principle in the history of networking — it's how TCP implementations from different vendors managed to interoperate in the 1980s, and it shaped four decades of protocol and API design. It is also, in the judgment of the IETF itself, a principle that curdles over time: RFC 9413 grew out of a draft bluntly titled "The Harmful Consequences of the Robustness Principle," arguing that liberal acceptance helps interoperability in the short term while quietly rotting the ecosystem in the long term.

Agent systems compress that long-term rot into weeks. When the "sender" is a language model emitting tool calls, every act of liberal acceptance — casting "5" to 5, dropping an unknown field, fuzzy-matching an enum typo — destroys the exact signal you need to keep the system healthy. The tool boundary is the one place in an agent architecture where failing loudly is the reliability feature, and most teams get it backwards.

A human engineer who sees a test fail on a change that couldn't possibly have caused it does something an agent cannot: they shrug. They hit rerun, mutter about the CI gods, and move on. That shrug encodes years of accumulated context — this test has been flaky since March, that service's staging environment falls over on Mondays, nobody trusts the WebSocket suite. A coding agent has none of this. It sees a red X and takes it as ground truth, because everything in its training and its prompt tells it that a failing test means the code is wrong.

What happens next is the expensive part. The agent doesn't shrug — it acts. It "fixes" code that was never broken. It reverts a correct change because the suite went red after applying it. It burns through its token budget chasing a phantom, adding retries and sleeps and defensive checks to code paths that were fine, until the flaky test happens to pass and the agent concludes its last mutation was the cure. Nondeterminism in your test substrate was always a tax on human attention. For agent loops, it's something worse: corrupted training signal, injected directly into the decision-making of a system that acts on it at machine speed.