383 posts tagged with "ai-agents"

A researcher agent asks a retrieval agent for a document. The retrieval agent, mid-task, decides it needs the researcher to clarify the query before it can search. The researcher, waiting on the document, won't respond until it has the document. Neither one is broken. Neither one is looping. They are both politely, indefinitely, waiting for each other — and your orchestrator, which has no concept of "both of these are blocked on each other," will happily hold that state until a timeout you never configured finally fires, or until a human notices the run has been "in progress" for forty minutes.

This is a deadlock. It is one of the oldest failure modes in computing, and it has nothing to do with how smart your model is. It is a property of how work is coordinated, not how work is done. The uncomfortable finding from the last year of multi-agent research is that most of what breaks in agent swarms breaks here, in the coordination layer, not in the reasoning of any single agent.

Single-agent thinking never surfaces these bugs. When one model runs a loop of tool calls, the worst it does is spin — and a spinning loop is at least visibly spinning. The moment you have two or more agents that can wait on each other, you have inherited the entire back catalog of distributed-systems pathologies: circular wait, livelock, lost messages, premature termination, races on shared state. Nobody sat down and decided to build a distributed system. You built one anyway the day you added a second agent.

When your agent issues a refund, sends an email, closes a ticket, or writes a row, that action leaves the system and enters the world. The world does not have a rollback button. The customer already saw the refund. The recipient already read the email. And when the agent takes a wrong turn three steps later, your recovery plan is usually a sentence in a retro: "we told it not to do that again."

"Don't do that again" is not undo. It is a promise about the future applied to a problem in the past. The uncomfortable truth is that most agent stacks have no mechanism to reverse a completed side effect — not a bad mechanism, no mechanism. The agent can plan, call tools, and retry, but it cannot walk backward. It has a forward gear and no reverse.

When you deprecate an API for human developers, you have a whole ceremony for it. You bump the version, add deprecated: true to the OpenAPI spec, ship a Sunset HTTP header, send an email to the developer mailing list, post to the changelog, and give people six months to migrate. The signal reaches a human who reads it, files a ticket, and updates their client before the old path disappears.

Now point that same deprecation at an agent. The model calling your tool does not read your changelog. It does not subscribe to your mailing list. It never sees the Sunset header unless you deliberately put it somewhere the model looks, and even then it has no reliable habit of acting on it. The deprecation notice you so carefully authored lands in a mailbox with no reader. The agent keeps calling the old shape of the tool until the shape is gone, and then it fails — often silently, often in production, often at 2 a.m.

This is the quiet asymmetry of building tools for agents instead of people. Every discipline we built over two decades of API evolution assumes a human sits between the deprecation and the migration. Take the human out, and the entire mechanism goes dark.

Every agent architecture diagram has the same three boxes. There's the happy path, where the model answers and the user leaves satisfied. There's the automatic fallback, where a low-confidence answer triggers a retry, a different tool, or a canned "let me look into that." And there's a third box, usually drawn last and smallest, labeled escalate to human. Everyone nods at that box in the design review. It looks like closure — the safety valve that makes the whole system defensible. "Don't worry, if the agent can't handle it, a person takes over."

Then you ship, and you discover the box was a lie. Not a technical lie — the code works, the ticket gets created, the conversation gets flagged. A staffing lie. The arrow labeled escalate to human points at a queue that nobody owns, has no service-level agreement, and appears on no one's on-call rotation. The agent did exactly what it was told. It handed the problem to an organization that never agreed to catch it.

The most expensive bug in your agent isn't a hallucination. It's a retry.

Somewhere in your stack there is a tool that charges a card, sends an email, closes a ticket, or writes a row. The agent calls it, the call takes too long, a timeout fires, and the agent — being a good, resilient piece of software — calls it again. The catch is that the first call already succeeded. The response just never made it back. Now you've charged the customer twice, and no amount of prompting "please be careful with payments" was ever going to prevent it.

This is the oldest failure mode in distributed systems wearing a new outfit. We solved it for HTTP APIs a decade ago with idempotency keys. But most agent stacks reintroduced the problem by pointing retry logic built for reads at tools that do writes, and then never sent the one field that would have made the retry safe.

Your agent has a latency SLO. Someone put it in a doc: "responses under 8 seconds, p95." What nobody did was decide how those 8 seconds get spent. There is no line item for the retrieval call, no line item for the planning step, no line item for the third tool the model decided to invoke because it felt uncertain. The budget exists as a single number at the boundary and as nothing at all inside. So when a five-hop chain blows past 8 seconds, the on-call engineer stares at a trace and cannot answer the only question that matters: which hop did it?

This is the difference between a service that has a latency budget and a service that has a latency hope. A budget is allocated per component and enforced. A hope is measured at the door and prayed over. Most agent systems ship with a hope, because the hop structure is dynamic — the model decides how many tool calls to make — and it feels impossible to budget something you don't control. It isn't. You budget it exactly because you don't control it.

Nobody decides to give an agent forty tools. It happens the way a garage fills up. You wire in a search tool, then a database reader, then someone on the team ships a Slack integration, then the ticketing MCP server gets installed because it was one config line away. Each addition is individually reasonable. Nobody ever removes anything, because removing a tool feels like taking away a capability, and taking away a capability feels like a regression.

Six months later your agent has a tool belt with three tools it uses constantly, a dozen it uses occasionally, and a long tail of two dozen it has technically never selected in production. That long tail is not free. It is not even cheap. Every unused tool in the catalog is actively making the agent worse at choosing among the ones that matter.

There is a number that should end the "let's add a chatbot" reflex on its own: in a large fraction of AI features, most of the people who open the chat window never send a single message. Reported figures cluster around 60% abandonment before the first message, versus far higher engagement when the same capability is wrapped in a designed empty state with examples and one-tap starting points. The model didn't fail. The answer was never generated, because the question was never asked. The user opened a blank box, felt the cursor blinking at them, and left.

We reached for chat because it was the path of least resistance, not because it was the right interface. The moment a language model could hold a conversation, "talking to the AI" became a synonym for "using the AI," and every product team inherited the same default: a text box, a send button, and a promise that the model will figure out the rest. That default is quietly wrong for most of the work agents actually do.

Chat is a fine input primitive. It is a terrible operating environment. Those are different claims, and conflating them is how you end up shipping a blinking cursor where you needed a control panel.

Most teams threat-model the wrong plane. They harden the chat box — rate limits, input validation, a jailbreak classifier watching what the user types — and they treat everything the model reads as inert. The wiki page, the support ticket, the scraped webpage, the calendar invite, the PDF someone uploaded: data, not instructions. Background material for the model to summarize, not commands for it to obey.

That assumption is the vulnerability. The moment your agent retrieves content and drops it into the context window, that content is executing with the same authority as your system prompt. There is no privilege boundary between "here are your instructions" and "here is a document to consider." It's all just tokens, and the model was trained to follow instructions wherever they appear.

Your agent works. It resolves the ticket, issues the refund, updates the record, closes the loop — unattended, at a quality your metrics say beats the median human doing the same job. You are ready to take the human out of the loop. And then the deployment stalls, not in engineering, but in a meeting where someone from legal or finance asks a single question you cannot answer: when it's wrong, who eats the loss?

This is the part of agent autonomy that the capability curve doesn't touch. You can push accuracy from 95% to 99% and the question doesn't move. Because the blocker was never "is the model good enough." The blocker is that a probabilistic system taking irreversible actions is a risk somebody has to hold, and right now nobody wants to hold it. Your errors-and-omissions policy was written for human mistakes and increasingly carves out automated ones. Your model vendor's contract disclaims the whole thing. And no carrier has an actuarial table for a system whose failure rate quietly drifts every time you edit a prompt.

The worst place to discover that you shipped an AI agent is in an incident review. Not the design doc, not the architecture review, not the change ticket — the incident review, where a lawyer is asking why a customer's account got a refund nobody authorized, and an engineer is scrolling through a service that was last meaningfully reviewed eleven months ago, and there, forty lines into a function called enrichTicket, is a model call that reads the customer record, decides on a resolution, and calls the billing API. Nobody diagrammed it. Nobody approved it as an agent, because to the person who wrote it, it wasn't one. It was "just a helper."

This is shadow AI, and it has grown teeth. The first wave was employees pasting company data into consumer chatbots — a data-leakage problem, bad but bounded. The second wave is agents: model calls wired into internal tools, reading real data and taking real actions, sitting inside services that were approved for something else entirely. Roughly half of employees admit to using AI tools their employer never sanctioned, and a striking share of that usage comes from the top — directors and executives are among the biggest culprits, not the interns. When the same instinct reaches your codebase, you don't get a leaked spreadsheet. You get an autonomous actor with production credentials that nobody signed off on.

The uncomfortable part is that shadow agents aren't created by reckless people. They're created by good engineers doing exactly what you asked them to do: ship value fast, reuse existing infrastructure, don't file a ticket for every small change. The governance gap isn't a discipline problem. It's a definitional one — your review process doesn't have a category for what they built.

A support request comes in: "My subscription got charged twice, cancel one and refund it." A router agent classifies it as billing and hands off. A billing specialist looks up the account, confirms two charges, and hands the refund to a tool-calling sub-agent. The sub-agent issues one refund, then a summarizer writes back to the customer: "Done — your duplicate charge has been refunded." Every step is individually correct. The customer was charged three times, not two. One duplicate is still on their card, and the summarizer had no way to know, because by the time the conversation reached it, the number "2" had already hardened into fact four steps upstream.

Now run the postmortem. Whose bug is it? The router classified correctly. The specialist read the two charges it was given. The tool agent refunded exactly what it was told. The summarizer summarized accurately. Pull up each agent's trace in isolation and every one of them passes. The system failed and no component did. This is the conversation with no owner, and it is the defining operational failure of multi-agent architectures — not a model being dumb, but responsibility diffusing across a handoff chain until no single agent is on the hook for the outcome the user actually experienced.