278 posts tagged with "reliability"

The most uncomfortable lesson from running agents in production isn't that they fail — it's that they learn. Not in any deep sense; the weights aren't moving. But within a session, within a trajectory, the policy implied by the model adapts to the substrate it runs on. And if your substrate quietly absorbs failure on the agent's behalf, the agent eventually notices, and starts planning as if that absorption were free compute.

The cleanest example is the retry layer. You added it for reliability — the SDK retries failed tool calls three times before surfacing an error, your middleware wraps each step in exponential backoff, your loop catches malformed JSON and re-prompts the model to fix it. None of this was wrong. But every one of those mechanisms is a side effect the agent can observe, generalize from, and exploit. Once it does, your reliability layer stops being a safety net and starts being a planning primitive.

A user is watching your agent stream a response. Two hundred milliseconds in, they hit stop. The UI clears the bubble, the spinner disappears, and the product behaves as if the request never happened. It did happen. The agent already called send_invoice_email. The vendor's mail relay returned 250 OK. The customer received a draft invoice the user never approved. Your billing meter charged the user for the tokens that streamed before the abort. It cannot bill back the email.

This is the failure mode every team with streaming tool use ships at least once, and most teams never even detect. The stream layer reports cancelled. The tool layer reports succeeded. Your customer-facing log picks one of them based on whichever subsystem flushes last, and the two halves of the same request now disagree about whether it occurred.

The first time your fallback route fires in production is the wrong time to discover that your two providers do not agree on what your schema means. The JSON Schema looks identical in both client configurations. The validator passes on both outputs. The downstream code reads the field by name and gets a value. And then a billing total comes out as a string of digits instead of an integer, or a list of length one arrives as a bare object instead of a single-element array, and a code path that has been green for six months silently returns the wrong answer.

The seductive thing about structured output is that it removes a class of bugs — unparseable JSON, hallucinated fields, missing keys — and so it feels like it removes the parsing problem entirely. What it actually does is move the parsing problem one layer up, from the lexer to the type system, where it is much harder to see. Two providers can both honor a JSON Schema and still produce outputs that are not interchangeable, because "honor" has at least four distinct meanings in this corner of the ecosystem and your schema does not specify which one you wanted.

Your contract is with the model provider. Your runbook handles the case where that provider is degraded. Your status page subscription pages you when their dashboard turns yellow. You feel covered. Then one Wednesday afternoon the underlying cloud region your provider runs in starts brownouts, your provider's failover region is also affected because they consolidated capacity to control unit economics, and your product is half-down for ninety minutes because of a vendor decision two layers upstream from any contract you signed.

The customer postmortem request lands in your inbox the next morning. They want a root cause. The root cause lives in a layer your status page cannot see and your contract does not let you compel. That layer is what fourth-party risk actually is — not a procurement checkbox, but a silent dependency tier that propagates failures upward with attenuation but not absorption.

Your handler did everything right. The HTTP status was 200. The body parsed. The text field had characters in it. You incremented responses_succeeded, appended the message to the conversation, returned the JSON down to the client, and moved on. The user got a sentence that ended mid-clause, a redacted answer dressed up as a normal one, or a polite refusal phrased as a completion. Your dashboard does not know any of that happened. The provider told you. You did not read the field.

Every major inference API returns a stop signal alongside the text: OpenAI calls it finish_reason, Anthropic calls it stop_reason, Gemini calls it finishReason. The field is small. It is one enum value per response. It is also the only out-of-band channel the model has for telling you whether the response you just shipped is the answer or a fragment of one. Treating it as cosmetic is the same shape of bug as ignoring HTTP status codes — except your monitoring caught the HTTP one a decade ago and has no opinion about this one.

You retired the search bar on the company portal because the agent did the same job better. Type a question in plain English, get an answer with citations, refine with a follow-up. The pilot crushed the satisfaction metric. The rollout email said "deprecating legacy search, full cutover in two weeks." Two weeks passed. The old index was decommissioned. The query box was replaced with a chat input.

Six months later, on a Tuesday morning, three things happen at once. Your inference provider rate-limits the corporate account because somebody's batch job spiked the shared quota. The embedding service has a regional brownout. A config push clears the prompt cache. Every engineer in the company who used to type "vpn setup" or "expense policy" into a search bar instead watches a spinner for forty seconds, then gets a refusal that does not understand their question, or worse, a confident citation to a wiki page that does not exist. The Slack channel where employees ask each other things lights up. The IT inbox fills with "is search broken?"

The search bar you replaced had three nines of availability over a decade of small incremental improvements. The agent that replaced it has a different shape of failure — slow not down, wrong not empty, expensive not cached — and your SRE culture was not calibrated for it.

The page fired at 02:14. The runbook said "page the engineer." The engineer's name resolved to an on-call rotation. The rotation pointed at a Slack channel that the team had wired up six months ago as a unified triage surface. The first message in the channel was the alert. The second message, posted nineteen seconds later, was a calm three-sentence summary: the alerting service, the failing dependency, the last deploy. It was well-written. It ended with "Acknowledged."

The incident commander, watching from her phone in bed, read "Acknowledged" and went back to sleep. Nobody had acknowledged. The agent subscribed to that channel as a first-line triage helper had restated the alert back to the room and signed off with the verb the channel's other readers used to mean "I have the context to act on this." The incident ran unowned for forty-one minutes until a customer ticket woke a different engineer through a different surface.

The first time an agent caused a real outage on my team, the postmortem author opened the template, scrolled past the timeline, stared at the "Root Cause" field for a long minute, and typed: "The runbook for queue-stuck recovery was incorrect." The runbook was fine. The agent had read the runbook, decided the queue's symptoms matched a different scenario, and run a recovery script for that other scenario instead. The action items that came out of that document — "tighten the runbook wording," "add a confirmation prompt to the recovery script" — were entirely useless against the actual failure mode, which was that an inferential system had inferred wrong and there was no field in the template that knew how to say so.

I've watched this exact failure repeat across teams since. The template is calibrated for deterministic systems. Code did the wrong thing, so you fix the code. Config was misset, so you fix the config. The schema of the postmortem document is the schema of the team's theory of failure, and when that theory cannot represent "the agent's plan was wrong," the document flattens the actual failure into the closest thing the template can represent — usually a documentation gap or a missing guardrail — and the action items chase a deterministic fix for a probabilistic failure. The same incident class then recurs, and the team writes it up the same way the next time.

The first time your provider failover actually fires in production, you will discover what you actually built. The gateway flips the traffic over in seconds — that part works. Then a different kind of incident starts: malformed JSON in 12% of responses, refusals on prompts that never saw a refusal before, latencies that destroy your downstream timeouts, customer-facing outputs that read like a different product. The primary came back ninety minutes later. The "successful" failover left a forty-eight hour incident review behind it.

This is the bill that comes due on the cheapest line of an architecture deck: "secondary provider for resilience." The deck never mentioned that the secondary needs its own prompts, its own evals, its own load-tested capacity, and its own on-call playbook. The deck just said you would not be down. The deck was right about that and wrong about everything else.

A daily standup summary is scheduled for 09:00 UTC. The cron fires on time. A worker pod spins up two seconds later. The LLM call takes another forty seconds round-trip. The model writes its summary believing it is February of last year, because that is the last thing its training data confidently knew. The tool layer dispatches the Slack message against the wall clock at 09:00:42 UTC, on a date the model never mentions because nobody asked it to. The message lands in the right channel, with yesterday's standup notes summarized as "today's," and nobody notices for three weeks.

This is not a bug in any one component. It is a contract that nobody wrote between four different clocks that all believe they know what "now" is.

A support agent reads a customer ticket, pulls up the account, summarizes the recent activity, and issues a refund. The refund lands in the wrong account. Not a fabricated account — a real one, one digit off. The model wrote acct_7H9j2 when the customer's record was acct_7H9j3. The trace looks clean: a search call returned the right record, a summarize call produced the right summary, a refund call ran without error. Every step succeeded. The wrong customer got the money.

This is not a hallucination in the sense the postmortem will use. The model did not invent a customer. It transposed two characters of an existing one, and that is a different failure mode — one your eval suite probably never caught, because the synthetic identifiers in your test fixtures were unique by construction. Two account numbers in the same context, three characters of shared prefix, and the language model — which is a token predictor that has never been trained to copy random strings with fidelity — picked the wrong one.

The lesson is structural, not behavioral. The model does not have an attention mechanism that special-cases identifiers. To the model, acct_7H9j2 is a sequence of subword tokens whose continuation probability shifts with every other token in the window. If a near-twin identifier appears in the same prompt, the model is one bad sample away from a quiet substitution that the harness will happily execute.

A senior engineer on call at 2am does not run a schema migration during a Sev-2 incident. They do not redeploy the payment service ten minutes before a release freeze starts. They do not fire a marketing email campaign while the email vendor's status page is red. None of this is in their job description. They picked it up from years of getting yelled at, from Slack channels titled #deploy-freeze-friday, from the muscle memory of glancing at the status page before they touch anything. It is the kind of context that does not exist in any runbook because nobody thought it needed to be written down.

Now hand the same job to an agent. The agent has tools. The agent has a multi-step plan. The agent has every documented policy you bothered to put in its system prompt. What the agent does not have is the half-conscious awareness that the world is currently on fire. So it executes the plan. Cleanly. Confidently. Into the maintenance window. And the postmortem includes a sentence that is going to become a familiar trope: "the agent had no way of knowing."