348 posts tagged with "observability"

A customer disputes a loan denial. A regulator opens an inquiry. A plaintiff's lawyer files discovery. All three arrive with the same deceptively simple question: what did your system decide, and why? The decision happened eighteen months ago. You pull up the case, and every component that produced the original output has since moved on. The hosted model version was deprecated and migrated. The system prompt has been edited nine times. The documents your agent retrieved were re-chunked, re-embedded, and re-ranked into a new index. And the sampling settings that made the whole thing non-deterministic were never logged in the first place.

You cannot reproduce the decision. Not because you were careless, but because nothing in your stack was built to be reproduced. Explainability-after-the-fact turns out to be a reproducibility problem in disguise — and reproducibility is something you either engineer in at decision time or lose forever.

The uncomfortable truth is that most teams discover this gap exactly when they can least afford to. The demand to reconstruct a decision almost never comes during normal operations. It comes attached to a lawsuit, an audit, or an angry customer with a regulator's phone number, and by then the window to have captured the right evidence closed a year and a half ago.

A researcher agent asks a retrieval agent for a document. The retrieval agent, mid-task, decides it needs the researcher to clarify the query before it can search. The researcher, waiting on the document, won't respond until it has the document. Neither one is broken. Neither one is looping. They are both politely, indefinitely, waiting for each other — and your orchestrator, which has no concept of "both of these are blocked on each other," will happily hold that state until a timeout you never configured finally fires, or until a human notices the run has been "in progress" for forty minutes.

This is a deadlock. It is one of the oldest failure modes in computing, and it has nothing to do with how smart your model is. It is a property of how work is coordinated, not how work is done. The uncomfortable finding from the last year of multi-agent research is that most of what breaks in agent swarms breaks here, in the coordination layer, not in the reasoning of any single agent.

Single-agent thinking never surfaces these bugs. When one model runs a loop of tool calls, the worst it does is spin — and a spinning loop is at least visibly spinning. The moment you have two or more agents that can wait on each other, you have inherited the entire back catalog of distributed-systems pathologies: circular wait, livelock, lost messages, premature termination, races on shared state. Nobody sat down and decided to build a distributed system. You built one anyway the day you added a second agent.

Every other change to your production system obeys a discipline. Code ships behind a flag, gets canaried to 1% of traffic, and rolls back in one click when a dashboard turns red. Schema migrations are staged and reversible. Even a CSS tweak goes through a pull request someone can read. Then there's the prompt. Someone edits a paragraph in a text box, clicks save, and the behavior of your product changes for every user at once — no canary, no diff anyone reviewed, no revert button that actually returns you to the prior state.

The uncomfortable part is that this isn't an oversight by careless teams. It's the default that the tooling produces. Prompts get filed under "configuration" because they're strings that live outside the compiled binary, and configuration has always been the thing you're allowed to change quickly without a full release. But a prompt isn't config. It's a program written in English, compiled by a nondeterministic interpreter you don't control, whose behavior you can only observe statistically. Treating it like a config value is the category error underneath a whole class of production incidents.

Every other line item on your infrastructure bill went through a gate. Someone filed a purchase order for the database cluster. Someone counted seats before buying the observability SaaS. Someone ran a capacity review before the team doubled its Kubernetes footprint. Then a model API showed up, and none of that happened.

An engineer added their API key to a config file. They wrote a create() call that looks exactly like every other function call in the codebase. It shipped. And the first time anyone in finance learned that this feature existed as a cost center was a variance line on the monthly invoice — a number nobody forecasted, nobody approved, and nobody can immediately explain.

This is the FinOps gap for AI, and it is not a monitoring problem. It is a governance problem wearing a monitoring costume. You can have perfect dashboards and still get surprised, because the spend was invisible to your approval process long before it was visible on a chart.

Your agent has a latency SLO. Someone put it in a doc: "responses under 8 seconds, p95." What nobody did was decide how those 8 seconds get spent. There is no line item for the retrieval call, no line item for the planning step, no line item for the third tool the model decided to invoke because it felt uncertain. The budget exists as a single number at the boundary and as nothing at all inside. So when a five-hop chain blows past 8 seconds, the on-call engineer stares at a trace and cannot answer the only question that matters: which hop did it?

This is the difference between a service that has a latency budget and a service that has a latency hope. A budget is allocated per component and enforced. A hope is measured at the door and prayed over. Most agent systems ship with a hope, because the hop structure is dynamic — the model decides how many tool calls to make — and it feels impossible to budget something you don't control. It isn't. You budget it exactly because you don't control it.

A support request comes in: "My subscription got charged twice, cancel one and refund it." A router agent classifies it as billing and hands off. A billing specialist looks up the account, confirms two charges, and hands the refund to a tool-calling sub-agent. The sub-agent issues one refund, then a summarizer writes back to the customer: "Done — your duplicate charge has been refunded." Every step is individually correct. The customer was charged three times, not two. One duplicate is still on their card, and the summarizer had no way to know, because by the time the conversation reached it, the number "2" had already hardened into fact four steps upstream.

Now run the postmortem. Whose bug is it? The router classified correctly. The specialist read the two charges it was given. The tool agent refunded exactly what it was told. The summarizer summarized accurately. Pull up each agent's trace in isolation and every one of them passes. The system failed and no component did. This is the conversation with no owner, and it is the defining operational failure of multi-agent architectures — not a model being dumb, but responsibility diffusing across a handoff chain until no single agent is on the hook for the outcome the user actually experienced.

Your eval dashboard is green. Turn-level accuracy sits at 95%, the LLM judge agrees with your annotators, and every regression test passes on the way to production. Then a user files a bug: the agent recommended a Postgres index on turn nine that directly contradicted the "we're on DynamoDB" constraint the user set on turn one. You pull the transcript. Every single turn, read in isolation, is a reasonable response. The conversation, read as a whole, is a disaster.

This is the central lie of turn-level evaluation. It grades request-response pairs because that's the unit that's cheap to label, and it quietly assumes that a conversation is just a bag of independent turns you can average. It isn't. The response at turn nine is conditioned on everything that came before it, and the failures that actually reach users almost never live inside a single turn — they live in the seams between turns, where state gets dropped, assumptions harden, and small errors compound into a wrong final answer.

The incident bridge is quiet in a way that means everyone is stuck. A support ticket says the agent told a customer their refund was approved when it wasn't. You have the full trace: the prompt, the retrieved account record, the tool calls, the model's reasoning, the final message. You replay it. The agent does the right thing. You replay it again. Right again. Nine times out of ten, the trace that produced the incident produces a correct answer instead. Someone on the call finally asks the question that the retro template can't handle: so what was the root cause?

There isn't one. Not in the sense the template means. The five-whys chain runs "the agent told the customer the wrong thing" → "because the model generated an approval" → "because it sampled a token sequence that asserted approval" → "because... that's what the probability distribution allowed." The last why bottoms out in a shrug. "The model sampled a bad token" is technically true and operationally useless. It names no fix, assigns no owner, and closes no gap. You can write it in the report, but everyone reading it knows you've documented a coincidence, not a cause.

You instrumented your agent the way every observability guide told you to. Every turn gets a span: the full prompt, the retrieved chunks, the model's chain of thought, the tool calls it considered, the plan it drafted and then abandoned, the final action. Storage is three cents a gigabyte and the traces have saved you a dozen times during incident response. You kept everything because keeping everything was free and occasionally heroic.

Here is the part nobody on the platform team modeled: that trace is written in fluent English, it speculates about your customers by name, and it drafts actions you never actually took. It is the single most quotable artifact your company produces, and a plaintiff's attorney will read the juiciest 200 words of it to a jury with your logo on the slide behind them. The reasoning trace that makes your system debuggable is the same reasoning trace that makes it liable.

Finance opens the invoice at the end of the month. One vendor. One number. It is bigger than last month, and it will be bigger next month. Then they ask the only question that matters — which feature spent this? — and nobody in the room can answer.

This is the quiet failure mode of running AI in production. Not that the bill is large; large is fine if the value is there. The failure is that the bill is unattributable. It arrives as a single line item — OpenAI, Anthropic, Bedrock, Azure — with none of the dimensions finance actually needs: not per feature, not per team, not per customer, not per successful task. You can see the total go up. You cannot see why, and by the time the invoice lands, the request context that could have explained it is long gone.

Your agent's internal metrics look great. Task completion is at 94 percent. The #agent-feedback Slack channel has been quiet for three weeks. Leadership is ready to turn it loose on customers. Then external users arrive, and within a month the numbers collapse: escalations spike, trust craters, and the postmortem question everyone asks is "how did dogfooding miss this?"

Dogfooding didn't miss it. Dogfooding hid it. Internal users are not a miniature version of your customer base — they are a population of expert operators who silently repair the agent's mistakes, learn which prompts to avoid, and share workarounds in DMs instead of filing bugs. Every repair makes the dashboard look better and the product worse-understood. The clean signal you're reading isn't quality. It's compensation.

Your cloud bill can tell you, down to the tag, that a forgotten S3 bucket cost $14,000 last month. Ask the same question of your LLM bill — which feature burned $40,000 in tokens — and the honest answer at most companies is a shrug. The provider invoice has one line per model per API key, three teams share the key, and the "AI costs" row in the finance spreadsheet is allocated by headcount, vibes, or whoever complained least in the last planning cycle.

This is not a small bookkeeping annoyance. When nobody can name the feature behind a dollar of token spend, two failure modes follow. Cheap features get throttled because they share a budget line with an expensive one. And genuinely wasteful features survive forever, because their cost is invisible — smeared across a shared key, a shared cache, and a shared agent loop that serves six different product surfaces.

The FinOps Foundation's 2026 survey found that 98% of organizations now actively manage AI spend, up from 63% a year earlier and 31% the year before — the fastest adoption curve the foundation has ever recorded. Everyone is suddenly doing "FinOps for AI." Far fewer have noticed that the core primitive of cloud FinOps — the resource tag — doesn't exist in the token world, and that three specific mechanics of modern LLM usage actively destroy attribution.