52 posts tagged with "distributed-systems"

If you lived through the microservices explosion of 2015–2018, the current state of MCP should feel uncomfortably familiar. A genuinely useful protocol appears. It's easy to spin up. Every team spins one up. Nobody tracks what's running, who owns it, or how it's secured. Within eighteen months, you're staring at a dependency graph that engineers privately call "the Death Star."

The Model Context Protocol is following the same trajectory, at roughly three times the speed. Unofficial registries already index over 16,000 MCP servers. GitHub hosts north of 20,000 public repositories implementing them. And Gartner is predicting that 40% of agentic AI projects will fail by 2027 — not because the technology doesn't work, but because organizations are automating broken processes. MCP sprawl is a symptom of exactly that problem.

Your monitoring dashboard is green. Response times look fine. Error rates are near zero. And yet your users are filing tickets about garbage answers, your agent is making confidently wrong decisions, and your support queue is filling up with complaints that don't correlate with any infrastructure alert you have.

Welcome to the unique hell of depending on an LLM API in production. It's an upstream service that can fail you while returning a perfectly healthy 200 OK.

Your agent worked perfectly in development. It answered test queries, called the right tools, and produced clean outputs. Then it hit production, and something went wrong on step seven of a twelve-step workflow. Your logs show the final output was garbage, but you have no idea why.

You add print statements. You scatter logger.debug() calls through your orchestration code. You stare at thousands of lines of output and realize you're debugging a distributed system with single-process tools. That's the fundamental mistake most teams make with AI agents — they treat them like programs when they behave like distributed systems.

Here is an uncomfortable fact about multi-agent AI systems: when you let two or more LLM-powered agents share resources and make decisions concurrently, they deadlock at rates between 25% and 95%. Not occasionally. Not under edge-case load. Under normal operating conditions with standard prompting, the moment agents must coordinate simultaneously, the system seizes up.

This is not a theoretical concern. Coordination breakdowns account for roughly 37% of multi-agent system failures in production, and systems without formal orchestration experience failure rates between 41% and 87%. The classic distributed systems failure modes — deadlock, livelock, priority inversion — are back, and they are wearing new clothes.

A multi-agent research tool built on a popular open-source stack slipped into a recursive loop and ran for 11 days before anyone noticed. The bill: $47,000. Two agents had been talking to each other non-stop, burning tokens while the team assumed the system was working normally. This is what happens when an agent pipeline has no backpressure.

The problem is structural. When an orchestrator agent decomposes a task into sub-tasks and spawns sub-agents to handle each one, and those sub-agents can themselves spawn further sub-agents or fan out across multiple tool calls, you get exponential work generation. The pipeline produces work faster than it can execute, finish, or even account for. This is the same problem that reactive systems, streaming architectures, and network protocols solved decades ago — and the same solutions apply.

You have three agents analyzing a customer support ticket. Two say "refund immediately," one says "escalate to fraud review." You pick the majority answer and ship the refund. Three days later, the fraud team asks why you auto-refunded a known chargeback pattern.

This is the consensus problem in multi-agent systems, and it turns out that distributed systems engineers solved important pieces of it decades ago. But naively transplanting those solutions — or worse, defaulting to majority vote — creates failure modes that are uniquely dangerous when your "nodes" are language models with opinions.

Three agents processed a customer account update concurrently. All three logged success. The final database state was wrong in three different ways simultaneously, and no error was ever thrown. The team spent two weeks blaming the model.

It wasn't the model. It was a race condition.

This is the failure mode that gets misdiagnosed more than any other in production multi-agent systems: data corruption caused by concurrent state access, mistaken for hallucination because the downstream agents confidently reason over corrupted inputs. The model isn't making things up. It's faithfully processing garbage.

Your agent is on step 7 of a 12-step workflow — it has already queried three APIs, written two files, and sent a Slack notification — when the process crashes. What happens next? If your answer is "restart from step 1," you're about to re-send that Slack message, re-write those files, and burn through your LLM token budget a second time. Databases solved this exact problem decades ago with write-ahead logging. The pattern translates to agent architectures with surprising fidelity.

The core insight is simple: before an agent executes any step, it records what it intends to do. Before it moves on, it records what happened. This append-only log becomes the single source of truth for recovery — not the agent's in-memory state, not a snapshot of the world, but a sequential record of intentions and outcomes that can be replayed deterministically.

Your agent processed a refund, but the response timed out. The framework retried. The customer got refunded twice. Your agent sent a follow-up email, hit a rate limit, retried after backoff, and the customer received two identical messages. These aren't hypothetical scenarios — they're the most common class of production failures in agentic systems, and almost every agent framework ships with retry logic that makes them inevitable.

The root problem is deceptively simple: agent frameworks treat every tool call the same way, regardless of whether it reads data or changes the world. A get_user_profile() call is safe to retry a hundred times. A send_payment() call is not. Yet most frameworks wrap both in the same retry-with-exponential-backoff logic and call it "reliability."

Your agent calls a tool. The tool times out. The agent retries. Each retry sends the full conversation context back to the LLM, burning tokens on a request that will never succeed. Meanwhile, the retry triggers a second tool call that depends on the first, which also fails and retries. Within seconds, a single flaky API has amplified into dozens of redundant requests, each one consuming compute, tokens, and time — and each one making the underlying problem worse.

This is the retry storm. It's not a new concept — distributed systems engineers have battled retry amplification for decades. But agentic AI systems make it dramatically worse in ways that microservice-era patterns don't fully address.

Every backend engineer knows that retries are essential. Every distributed systems engineer knows that retries are dangerous. When you put an LLM agent in charge of retrying tool calls, you get both problems at once — plus a new one: every retry burns tokens. A single flaky API endpoint can turn a $0.01 agent task into a $2 meltdown in under a minute.

The retry storm problem isn't new. Distributed systems have dealt with thundering herds and cascading failures for decades. But agentic systems amplify the problem in ways that microservice patterns don't fully address, because the retry logic lives inside a probabilistic reasoning engine that doesn't understand backpressure.

When a multi-agent pipeline starts producing garbage outputs, the instinct is to blame the model. Bad reasoning, wrong context, hallucination. But in practice, a large fraction of multi-agent failures trace back to something far more boring: agents that can't reliably communicate with each other. Malformed JSON that passes syntax validation but fails semantic parsing. An orchestrator that sends a task with status "partial" that the downstream agent interprets as completion. A retry that fires an operation twice because there's no idempotency key.

These aren't model failures. They're interface failures. And they're harder to debug than model failures because nothing in your logs will tell you the serialization contract broke.