65 posts tagged with "distributed-systems"

A researcher agent asks a retrieval agent for a document. The retrieval agent, mid-task, decides it needs the researcher to clarify the query before it can search. The researcher, waiting on the document, won't respond until it has the document. Neither one is broken. Neither one is looping. They are both politely, indefinitely, waiting for each other — and your orchestrator, which has no concept of "both of these are blocked on each other," will happily hold that state until a timeout you never configured finally fires, or until a human notices the run has been "in progress" for forty minutes.

This is a deadlock. It is one of the oldest failure modes in computing, and it has nothing to do with how smart your model is. It is a property of how work is coordinated, not how work is done. The uncomfortable finding from the last year of multi-agent research is that most of what breaks in agent swarms breaks here, in the coordination layer, not in the reasoning of any single agent.

Single-agent thinking never surfaces these bugs. When one model runs a loop of tool calls, the worst it does is spin — and a spinning loop is at least visibly spinning. The moment you have two or more agents that can wait on each other, you have inherited the entire back catalog of distributed-systems pathologies: circular wait, livelock, lost messages, premature termination, races on shared state. Nobody sat down and decided to build a distributed system. You built one anyway the day you added a second agent.

When your agent issues a refund, sends an email, closes a ticket, or writes a row, that action leaves the system and enters the world. The world does not have a rollback button. The customer already saw the refund. The recipient already read the email. And when the agent takes a wrong turn three steps later, your recovery plan is usually a sentence in a retro: "we told it not to do that again."

"Don't do that again" is not undo. It is a promise about the future applied to a problem in the past. The uncomfortable truth is that most agent stacks have no mechanism to reverse a completed side effect — not a bad mechanism, no mechanism. The agent can plan, call tools, and retry, but it cannot walk backward. It has a forward gear and no reverse.

The most expensive bug in your agent isn't a hallucination. It's a retry.

Somewhere in your stack there is a tool that charges a card, sends an email, closes a ticket, or writes a row. The agent calls it, the call takes too long, a timeout fires, and the agent — being a good, resilient piece of software — calls it again. The catch is that the first call already succeeded. The response just never made it back. Now you've charged the customer twice, and no amount of prompting "please be careful with payments" was ever going to prevent it.

This is the oldest failure mode in distributed systems wearing a new outfit. We solved it for HTTP APIs a decade ago with idempotency keys. But most agent stacks reintroduced the problem by pointing retry logic built for reads at tools that do writes, and then never sent the one field that would have made the retry safe.

Your agent has a latency SLO. Someone put it in a doc: "responses under 8 seconds, p95." What nobody did was decide how those 8 seconds get spent. There is no line item for the retrieval call, no line item for the planning step, no line item for the third tool the model decided to invoke because it felt uncertain. The budget exists as a single number at the boundary and as nothing at all inside. So when a five-hop chain blows past 8 seconds, the on-call engineer stares at a trace and cannot answer the only question that matters: which hop did it?

This is the difference between a service that has a latency budget and a service that has a latency hope. A budget is allocated per component and enforced. A hope is measured at the door and prayed over. Most agent systems ship with a hope, because the hop structure is dynamic — the model decides how many tool calls to make — and it feels impossible to budget something you don't control. It isn't. You budget it exactly because you don't control it.

Pull up your request logs from the last time you hit sustained 429s. You will probably find something odd: the errors don't arrive as a steady stream. They arrive in waves — a burst of 429s, a quiet gap, a bigger burst, another gap. The provider's quota didn't change between waves. Your traffic didn't spike. What you are looking at is your own retry logic, synchronized against itself. Every client that failed at second zero computed the same backoff delay, slept the same duration, and woke up at the same instant to fail together again.

This is the thundering herd, and the punchline is that the standard fix — exponential backoff — does not prevent it. Deterministic exponential backoff organizes the herd. It takes a crowd of clients that failed at roughly the same moment and marches them forward in lockstep: everyone retries at 1 second, then everyone at 2, then everyone at 4. The load spikes get farther apart, but each spike is just as tall as the first. If the spike is what triggered your rate limit, you have built a metronome that re-triggers it forever.

Your agent died at step seven of a twelve-step task. The framework caught the exception, waited with exponential backoff, and retried. It retried the step — with the same context window that had accumulated three failed tool calls, a half-parsed error message, and a plan the model had already abandoned. The retry failed too, of course, because a retry is a bet that the world changed, and nothing about that agent's world had changed. What needed to change was the agent's state — and no retry policy in any agent framework makes that decision.

Erlang's OTP libraries codified this exact decision thirty years ago, for telephone switches that had to run for decades. The insight behind supervisor trees was never "restart things when they crash." It was that how to recover is a separate concern from doing the work, owned by a separate process, arranged in a hierarchy where each level knows a little more about what recovery means. Most agent frameworks today bolt retries onto individual calls, which is like putting a try/catch around every line of a telephone switch. What they need is the hierarchy.

We spent two decades teaching services to retry safely. The playbook is well worn: a client generates a unique idempotency key, attaches it to the request, and the server records the key alongside the result inside the same transaction that performs the work. A dropped connection, a timeout, a 500 — the client retries with the same key, the server recognizes it, and returns the recorded result instead of charging the card twice. Stripe shipped this pattern years ago and it became table stakes for any API that touches money.

That entire design rests on one assumption nobody wrote down: the caller repeats its request byte-for-byte. The retry carries the same key because the retry is the same code path re-executing with the same variables. Break that assumption and the whole scheme quietly stops working.

There is a class of agent bug that does not appear in any single component when you look at it in isolation. The model is fine. The tool is fine. The retry policy is fine. The timeout values are even, on paper, generous. And yet a tool that consistently completes in eight seconds keeps landing against an agent that has already declared it a failure at seven point nine, replanned around an "error" that never happened, and started a second call that the first call's result is about to collide with.

The bug is not in any of the boxes. It is in the gap between two clocks that nobody agreed should be the same clock.

The clearest sign that an agent's session model is broken is when a tool result has nowhere to go. The agent fired a long-running call — a render, a provisioning job, a multi-step query. The user watched the spinner for a few seconds, decided they didn't need it after all, closed the tab, and moved on. Forty seconds later the tool finishes. Its callback hits your gateway with a conversation_id that no longer points at anything. The gateway has two equally bad options: silently drop the result, or stitch it into whatever session inherits that ID next.

Most teams discover this failure mode the same way: a support ticket where a user sees an answer they did not ask for, attached to a conversation they did not start. Or a downstream system that processed the same charge twice because the gateway helpfully "retried" delivery against the next active session. Or — most commonly — nothing visible at all, just a slow drift in completion metrics that nobody can correlate to anything specific, because the failures don't fire alerts; they fire emptiness.

The clearest sign that an agent's tool-call abstraction is broken is when the trace shows the step marked done and the downstream system shows nothing happened. The model called a tool, received a job ID back, treated the job ID as the answer, and moved on. Three minutes later the actual work either succeeded with nobody listening or failed with the error landing in a log nobody reads. The user sees a confident summary; the operations queue sees a stranded task.

This is the failure mode the function-calling abstraction quietly enables. JSON schemas describe parameters and return types, but they do not distinguish between "this tool returns a result" and "this tool returns a receipt for an operation whose result you will need to ask about later." The model treats both the same way, because to the planner they look the same — a successful tool call with a non-error payload.

The pager went off at 14:02 UTC. Not for latency, not for errors — for spend. The cost dashboard showed a vertical line: three minutes of input-token billing at roughly nine times the trailing hourly average, then back to normal. No regression had shipped. No tenant had onboarded. Traffic was flat to the minute. The only thing that changed is that a single prompt prefix — the 14K-token system message that every agent in the fleet shared — had quietly expired on the provider side, and a thousand workers had all decided, within the same 200ms window, that they were the ones who needed to write it back.

This is a cache stampede. It is the same bug operators have been writing post-mortems about since memcached shipped in 2003. What is new in 2026 is that the cache it stampedes is no longer yours. It lives inside your model provider, you cannot inspect its state, and every miss costs real money instead of a few extra database queries. The synchronization bug that database engineers learned to jitter away two decades ago has quietly reappeared on a bill line item nobody thought to defend.

Agent A asks Agent B for a piece of data it needs to finish its task. Agent B, before answering, asks Agent A for a piece of context it needs to produce that data. Both requests cross a "human review required" boundary on the way out. The first request lands in a Slack approval channel watched by Priya. The second lands in a Jira queue watched by Marcus. Priya is at lunch. Marcus is in a customer call. Neither knows the other exists. The workflow hangs for nineteen hours, and nobody notices until a customer escalation forces somebody to ask why the rollup never landed.

This is not a novel failure. It is the oldest failure in distributed systems, wearing a new costume. The Coffman conditions — mutual exclusion, hold and wait, no preemption, circular wait — were named in 1971, and a multi-agent system with human-in-the-loop approval queues satisfies all four by default. The new wrinkle is that one of the "resources" in the deadlock is a person's attention, which means your liveness guarantee is now bound by how quickly two humans who don't know they're paired can independently context-switch.