Skip to main content

The Webhook Your Agent Sent That Another Team's Agent Received

· 10 min read
Tian Pan
Software Engineer

The first time two agents from different teams started talking to each other, nobody wrote a feature flag for it. The conversation just appeared in the logs. A support agent published an event called ticket.resolved to the shared bus. A growth agent, listening for ticket.resolved from a totally different product line, picked it up, congratulated the (wrong) customer in a follow-up email, and offered a discount on a product they did not own. By the time anyone noticed, the loop had run forty-three times.

The shared bus did exactly what shared buses do. It routed messages by topic name to anyone who subscribed. It did not know that one team meant ticket.resolved to refer to a payments support flow and the other team meant it to refer to a returns flow. It did not know that the second agent was authorized to act on the first agent's events. It did not, in fact, know that either side was an agent at all. The bus saw bytes labeled with a string and delivered them. The agents saw a message that matched their schema and acted.

This is the failure mode I want to name: accidental cross-agent communication. Two systems that were never designed to talk start trading messages because the fabric beneath them has no notion of which agent owns which topic, no notion of which agent is allowed to consume which event, and no notion of provenance. The cost of being wrong about routing used to be a stuck queue or a dropped notification. The cost now is an autonomous loop that runs until somebody pulls the cord.

Why the bus has no idea who is talking

Most event buses were designed for services, not agents, and the difference matters. A service that subscribes to a topic does one specific thing with the payload: write to a database, fire a side effect, emit a metric. A reviewer looking at that subscription can see, in code, exactly what the consumer will do with the message. Static review catches the obvious mistakes before they ship.

An agent that subscribes to a topic does whatever its prompt and tools say it should do, which depends on the content of the message itself. The consumer's behavior is no longer in code. It is in the message, the system prompt, the tool registry, and the LLM's interpretation of all three. The reviewer cannot, in advance, know what the agent will do with ticket.resolved from an unfamiliar publisher, because the answer is a function of a string the publisher controls.

This collapses a load-bearing assumption of pub/sub design. Topics were a coordination mechanism — a way for unrelated systems to agree on what an event meant. They became, for agents, an instruction mechanism — a way for an unrelated system to nudge another system into action. That shift is small in code and enormous in consequence.

Pub/sub buses do not enforce topic ownership by default. Kafka's multi-tenancy guide acknowledges this directly: you need namespace prefixes (finance.transactions, marketing.events) and ACLs to keep tenants from stepping on each other, and even with those in place, a misconfigured consumer group can subscribe across boundaries. The bus is a postal system. Anyone who knows the address can read what arrives.

The four ways the cross-talk actually starts

In the postmortems I have read or written, the first cross-team agent contact is almost never deliberate. It happens in one of four ways, and the patterns repeat enough to be worth naming.

Generic topic names. Two teams independently decide their domain event is called task.completed, order.updated, or user.signup. They are talking about different tasks, orders, and users, but they are speaking on the same channel. As long as both teams stay inside their own service mesh, this is a latent bug. The moment an agent subscribes with a broad filter, the bug becomes a feature, and the agent starts acting on the wrong domain.

Wildcard subscriptions. Agents that need broad situational awareness often subscribe with patterns like *.failed or support.*. The wildcard is convenient: the agent learns about failures it did not know to expect. It also catches every event from every other team that happens to land in the matching namespace. A library like agent-event-bus makes wildcard topic matching a first-class feature precisely because agents want it. The cost of that convenience is paid the first time a team you have never met publishes something to inventory.failed and your reconciliation agent decides to reconcile it.

Topic reuse after acquisitions or reorgs. A team that owned a topic gets dissolved. Their consumers go away. Six months later, a new team picks the same topic name for a brand-new flow, not realizing the old publishers are still emitting. The agent on the new flow starts receiving stale events from a system nobody remembers. The old publisher's owner is no longer at the company. The fix is archaeological.

Schema convergence without coordination. Two teams independently arrive at JSON shapes that happen to match: both have { id, status, customer_id, timestamp }. Both publish under similar-sounding topic names. The bus delivers the wrong payloads to the wrong agents, and the agents accept them because the schema matches. The validator never fires. The agent acts on coherent garbage.

Each of these patterns predates agents. What is new is the consequence. A misrouted webhook used to mean a 404 in a log file. A misrouted event to an agentic consumer means an action — a refund, an email, a database write, a downstream tool call — performed on behalf of a system that did not authorize it.

The authorization gap

The deeper problem is that the message bus has nothing to say about authorization. When two services trade messages, authorization is enforced at the action layer: even if Service A receives an event meant for Service B, Service A still has to call an authorized API to do anything with it, and that API can check the caller's identity. The blast radius is bounded by what Service A was allowed to do anyway.

When an agent receives a misrouted event, the action layer is the agent's tool registry, and the agent's tool registry was scoped to do a great many useful things. The agent that listens for ticket.resolved can send emails, issue refunds, update CRM records, and call the billing API — all things it is supposed to do, in the right context. There is nothing on the receiving end that distinguishes "I received this event from my own team's publisher" from "I received this event from a stranger." The same tool calls fire either way.

Recent work on agent identity is direct about the gap. OAuth and OpenID Connect were designed for users delegating to applications, not for agents delegating to sub-agents or accepting work from peers. There is no widely deployed protocol for an agent to ask "who published this message, and are they allowed to ask me to do anything?" The proposed solutions — agent identity URIs, OIDC extensions for agents, capability tokens — are early. In the meantime, the bus is authenticating the publisher to the bus and authorizing nothing downstream of that.

This is the part that should make you uncomfortable. The classic security model assumed that if you locked the doors of every service, you locked the building. Agents are users with credentials, walking from room to room based on what they overhear. If the rooms keep no record of who is allowed to overhear what, the building has no security model. It has hallways.

What containment actually looks like

The fix is not "more careful naming." Naming conventions are necessary and not sufficient; people will violate them, especially under deadline. The fix is to make the bus itself aware of which agent owns which topic, and to give every consumer a way to verify provenance before acting.

A workable containment pattern has four pieces. None of them are novel, and that is the point — the failure mode persists because teams treat the bus as a neutral substrate rather than as a place where authorization decisions must be made.

The first piece is prefix-and-ACL discipline at the broker level, not at the convention level. Every topic name carries a team prefix (payments.support.ticket.resolved), and the broker enforces that only the owning team's principals can publish to that prefix and only allow-listed principals can subscribe to it. Wildcard subscriptions across prefixes require a written sign-off and an explicit grant. The ACL is the system of record; the README is a courtesy.

The second piece is signed publisher identity in the envelope, not in the topic name. Every event carries a verifiable claim about who published it, signed with a key the consumer can validate. The consumer's first check, before the agent ever sees the payload, is "is this from a publisher I am authorized to act on?" If the answer is no, the message is dropped at the edge, before the LLM is invoked. This is the agent equivalent of webhook signature validation, and the same arguments that justify HMAC on webhooks justify signatures on intra-bus messages once agents are the consumers.

The third piece is agent-side allow-lists for source domains. The agent's configuration declares which publishers it is willing to act on. A support agent for payments accepts events from payments.support.*. It rejects everything else, including events that happen to match its schema. This makes accidental cross-talk into an explicit configuration change, which is reviewable and auditable.

The fourth piece is action-level provenance checks. When the agent decides to call a tool — issue a refund, send an email — the tool call carries the provenance of the originating event. The tool gateway can refuse to act on behalf of an event whose publisher is outside the agent's authorized domain. This is the belt to the suspenders: even if a misrouted event gets through, the tool refuses to fire.

You will notice that this is more or less the same architecture that mature payment systems use to keep money from moving in unintended ways. The bus is a place where authorization is enforced, not just a place where messages are delivered. Until that is true for the agent fabric, the next forty-three-iteration loop is already on its way.

The thing to internalize

The shared event bus is one of the most useful design patterns we have. It is also a default-open communication fabric, and we have spent the last decade adding agents to it without changing the defaults. The webhook your agent sent that another team's agent received is not a freak event. It is the natural behavior of a substrate that routes by name and authorizes nothing, populated by consumers whose behavior is a function of the content of every message they receive.

The teams that will not be writing this postmortem in six months are the ones treating the bus as a privileged surface — the same way they treat the database, the secrets store, and the production API. Topic ownership belongs in the broker. Publisher identity belongs in the envelope. Source allow-lists belong in the agent. Provenance belongs in the tool call. Once those four are in place, accidental cross-agent communication stops being a loop and starts being a rejected message in a log file, which is exactly what it should have been all along.

References:Let's stay in touch and Follow me for more thoughts and updates