96 posts tagged with "architecture"

A support request comes in: "My subscription got charged twice, cancel one and refund it." A router agent classifies it as billing and hands off. A billing specialist looks up the account, confirms two charges, and hands the refund to a tool-calling sub-agent. The sub-agent issues one refund, then a summarizer writes back to the customer: "Done — your duplicate charge has been refunded." Every step is individually correct. The customer was charged three times, not two. One duplicate is still on their card, and the summarizer had no way to know, because by the time the conversation reached it, the number "2" had already hardened into fact four steps upstream.

Now run the postmortem. Whose bug is it? The router classified correctly. The specialist read the two charges it was given. The tool agent refunded exactly what it was told. The summarizer summarized accurately. Pull up each agent's trace in isolation and every one of them passes. The system failed and no component did. This is the conversation with no owner, and it is the defining operational failure of multi-agent architectures — not a model being dumb, but responsibility diffusing across a handoff chain until no single agent is on the hook for the outcome the user actually experienced.

You wired up a second model vendor because the first one went down. It was a Tuesday, the primary provider's status page was a wall of red for a few hours, and your feature was dead in the water with it. So you did the responsible thing: you added a fallback. Now, if OpenAI is unreachable, you route to Anthropic. If Anthropic rate-limits you, you fall back to Gemini. The architecture diagram looks clean and grown-up. Reliability, solved.

Except you didn't add a replica. You added a second opinion. And a second opinion is a very different thing to operate than a second copy.

The mental model you imported comes from the stateless-service playbook: run three identical instances behind a load balancer, and if one dies the other two serve the exact same responses. That works because the replicas are byte-for-byte interchangeable. Two language models from two vendors are not. They were trained on different data, tuned with different objectives, and they disagree — systematically, not randomly — on exactly the inputs where your users notice.

Here is a migration story that repeats in every team shipping agents. You built your agent on one provider's SDK. Tool definitions live as JSON schemas in the exact shape that provider expects. Tool results get formatted into that provider's message structure. Then something forces a change — a better model ships from a competitor, procurement mandates a second provider for redundancy, an MCP server replaces a hand-rolled integration — and you discover the real inventory of the migration: it isn't one API client. It's every tool definition, every result formatter, every retry handler, and every test fixture in the codebase.

The failure isn't that you chose the wrong provider. It's that you let someone else's serialization format become your internal architecture. There is a twenty-year-old answer to exactly this problem — Alistair Cockburn's hexagonal architecture, better known as ports and adapters — and agent systems are the most compelling new use case it has had in years.

Somewhere in your codebase there is a gateway that lets you swap claude-sonnet for gpt-5 by changing one string. Your architecture review praised it. Your CTO sleeps better because of it. And it is quietly costing you 90% cache discounts, schema-enforced outputs, and the reasoning-effort knobs that separate a great production model from a mediocre one.

That is the unadvertised price of the unified LLM API. Every abstraction layer that promises "swap providers in one line" delivers that promise by projecting every provider onto the subset of features they all share — and the features that fall outside that subset are precisely the ones where providers compete hardest. Prompt caching semantics, structured output enforcement, extended thinking budgets, server-side tool execution: these are the levers that determine your real cost and quality curve, and your gateway may be silently dropping them on the floor.

Your team rejected the microservices split in 2024. There was a two-hour meeting, a heated Slack thread, and a spreadsheet comparing operational costs. The conclusion was firm: the monolith stays until the platform team ships multi-region deploys. Everyone who was in the room still remembers.

Your coding agent was not in the room. Last Tuesday it proposed the split again — confidently, with a clean migration plan and a well-argued design doc. The Tuesday before that, a different agent session proposed extracting the billing service. Next week, a third one will suggest replacing your bespoke job queue with the message broker you evaluated and rejected twice. None of them are wrong on the merits as they can see them. They just can't see the merits, because the reasoning that settled these questions lives in a Slack thread that expired, a meeting that wasn't recorded, and the heads of two engineers, one of whom left.

This is the quiet failure mode of agent-assisted engineering: settled questions get re-litigated at machine speed. And the fix is a practice most teams filed under bureaucratic nice-to-haves a decade ago — the architecture decision record.

If your agent team coordinates through a shared plan file, a repo, or a design doc that everyone reads and writes, congratulations: you have reinvented the blackboard architecture. It was state of the art in 1975. The uncomfortable part is not the reinvention — good ideas deserve to come back. The uncomfortable part is that the original had three load-bearing components, and most modern agent stacks rebuilt only one of them.

Hearsay-II, the DARPA-funded speech understanding system built at Carnegie Mellon between 1971 and 1976, faced a problem that should sound familiar: many specialized, unreliable experts — acoustic analyzers, syntax predictors, semantic raters — none of which could solve the problem alone, all of which needed to build on each other's partial guesses. The architecture that emerged had a shared workspace (the blackboard), independent specialists (knowledge sources), and a scheduler that decided, at every step, which specialist's contribution was worth executing next. Fifty years later, teams wiring LLM agents together are converging on the same shape — a lead agent, a set of workers, a shared artifact — and hitting failure modes the blackboard literature named and solved before most of us were born.

Your agent died at step seven of a twelve-step task. The framework caught the exception, waited with exponential backoff, and retried. It retried the step — with the same context window that had accumulated three failed tool calls, a half-parsed error message, and a plan the model had already abandoned. The retry failed too, of course, because a retry is a bet that the world changed, and nothing about that agent's world had changed. What needed to change was the agent's state — and no retry policy in any agent framework makes that decision.

Erlang's OTP libraries codified this exact decision thirty years ago, for telephone switches that had to run for decades. The insight behind supervisor trees was never "restart things when they crash." It was that how to recover is a separate concern from doing the work, owned by a separate process, arranged in a hierarchy where each level knows a little more about what recovery means. Most agent frameworks today bolt retries onto individual calls, which is like putting a try/catch around every line of a telephone switch. What they need is the hierarchy.

Ask a coding agent to build a React component and it writes idiomatic, hook-shaped, accessibility-annotated code on the first try. Ask the same agent to use your in-house ORM — the one your platform team has maintained for six years, the one with excellent docs and a hundred internal consumers — and it hallucinates methods that don't exist, invents configuration options from some other library, and confidently ships code that compiles against an API it made up.

The difference isn't quality. Your ORM might be better-designed than half the open-source libraries the model handles flawlessly. The difference is training data. React has millions of public repositories behind it; your framework has zero. In the vocabulary of natural language processing, your internal framework is a low-resource language — and every consequence NLP researchers documented for low-resource languages now applies to your codebase.

Eighteen months ago, "we built our own retrieval pipeline" was a reasonable thing to say in an architecture review. You had a chunking strategy, an embedding model you'd benchmarked, a vector store you'd tuned, a re-ranker, and a context-packing heuristic that took three engineers a quarter to get right. That stack was a real piece of differentiated infrastructure. Today, the same capability is a single hosted tool call: upload files to a vector store, attach it to a request, and the provider parses, chunks, embeds, stores, retrieves, and re-ranks — all behind one API. The three-engineer-quarter is now a config object.

This is the uncomfortable pattern of building AI products right now. The line between what you build and what you buy is not fixed. It moves — and it moves in one direction. Every few months a provider ships a primitive that absorbs a layer you used to own: memory, retrieval, structured output, tool routing, even multi-step orchestration. The infra you were proud of last quarter becomes the thing a competitor gets for free this quarter, with better defaults and a lower latency floor because it runs inside the provider's own datacenter.

The instinct is to treat this as a threat to defend against. That's the wrong frame. The provider absorbing commodity infra is doing you a favor — it deletes maintenance you never wanted. The real question is whether the thing you chose to build sits above or below the rising waterline. Most teams never make that call explicitly. They build whatever the demo needed, and then discover a year later that 70% of their codebase is reimplementing a primitive the provider now offers, while the 30% that's actually defensible got starved of attention.

The post-mortem read like a hallucination report. A user had acted on a confidently-worded recommendation that turned out to be wrong in a way the model would not have written if it had finished — except the trace showed the model had not finished. The provider connection dropped at token 412 of an expected 800. The client's error handler logged the failure. The persisted partial message, written to the conversation history as tokens arrived, sat in the user's UI looking exactly like every other complete answer. They acted on it. Support categorized the ticket as a content-quality issue. It took two weeks to route it to the platform team.

Nothing in this chain was a model failure. The model behaved correctly for the 412 tokens it produced. The failure was that the streaming UI and the durable conversation history had quietly disagreed about what counts as a message — and during the exact failure mode that streaming was supposed to make tolerable, the disagreement became the canonical record.

This is the contract between optimistic rendering and durable storage. Most chat products inherit it from a tutorial or a framework without thinking about it as a contract at all, and the gap shows up as a tail of incidents that look like model bugs and aren't.

A support agent reads a customer ticket, pulls up the account, summarizes the recent activity, and issues a refund. The refund lands in the wrong account. Not a fabricated account — a real one, one digit off. The model wrote acct_7H9j2 when the customer's record was acct_7H9j3. The trace looks clean: a search call returned the right record, a summarize call produced the right summary, a refund call ran without error. Every step succeeded. The wrong customer got the money.

This is not a hallucination in the sense the postmortem will use. The model did not invent a customer. It transposed two characters of an existing one, and that is a different failure mode — one your eval suite probably never caught, because the synthetic identifiers in your test fixtures were unique by construction. Two account numbers in the same context, three characters of shared prefix, and the language model — which is a token predictor that has never been trained to copy random strings with fidelity — picked the wrong one.

The lesson is structural, not behavioral. The model does not have an attention mechanism that special-cases identifiers. To the model, acct_7H9j2 is a sequence of subword tokens whose continuation probability shifts with every other token in the window. If a near-twin identifier appears in the same prompt, the model is one bad sample away from a quiet substitution that the harness will happily execute.

The most expensive bug in an agent system is the one with no error message. Worker proposes a draft. Verifier rejects it with a paragraph of feedback. Worker revises. Verifier rejects again. The loop keeps spinning, the trace keeps growing, the bill keeps climbing, and from the outside the system looks like it is working — diligently, in fact, because both models are doing their assigned job. What nobody priced in is that the verifier's acceptance criteria are not fixed across calls. The target the worker is chasing is moving, and the loop has no convergence guarantee.

You shipped "iterate until satisfied," and you shipped a search through a space whose extrema may not exist.