238 posts tagged with "reliability"

Watch an agent reason through a multi-step task and you will notice something familiar: it plans the way you debug. Try an approach, look at the result, and if it is wrong, back out and try another. The agent talks about its plan as a tree of options it can explore, prune, and revisit. That mental model is correct inside a code sandbox, where every action has an implicit undo. It is dangerously wrong the moment the agent touches the world.

A sent email does not unsend. A charged card does not uncharge without a refund flow, a fee, and a customer who already saw the notification. A deleted row is gone unless someone wired up soft deletes. A posted Slack message has already been read. The agent's planning model has no native concept of the one-way door — the action that, once taken, removes the option of pretending it never happened.

This is not a model intelligence problem. A smarter model still does not know which of your tools is reversible, because reversibility is not a property of the action. It is a property of the system the action lands in. You have to tell it.

A user asks your agent what the current price tier is for an enterprise plan. The agent retrieves a chunk, reads it, and answers: "$2,000 per month." Confident, sourced, formatted nicely. The problem is that pricing changed four days ago. The number the agent quoted was true last week. The chunk it retrieved was embedded before the change, and the index has not caught up.

Nobody decided this would happen. There was no design review where someone said "the agent may answer from data up to four days old." There is just a re-indexing job that runs nightly, or weekly, and a content team that edits the source whenever they feel like it, and a gap between those two clocks that nobody measures. That gap is a service level objective. It exists whether or not you wrote it down. The only question is whether you set it on purpose or inherited it by accident.

Every agent architecture diagram has a box labeled "escalate to human." It is drawn with a clean arrow, it satisfies the reviewer, and it makes the system feel safe. What the diagram never shows is the person on the other end of that arrow — whether they exist, whether they are awake, and whether they will answer before the agent's patience runs out.

Human-in-the-loop is sold as a design pattern. In production it behaves like a staffing problem. The pattern assumes a human is standing by; the staffing reality is that escalations do not arrive when humans are available — they arrive on their own schedule. A burst at 2am when an overnight batch job trips a guardrail. A long tail through lunch when half the reviewers are away from their desks. A steady drip that quietly outgrows the two-person team that looked sufficient during the demo, when the agent handled ten requests a day instead of ten thousand.

The gap between "we have an escalation path" and "escalations get answered" is where agentic systems fail in ways no eval catches. The eval measures whether the agent escalates correctly. It never measures whether anyone was there.

An agent books the wrong flight. It sends an apology email to the wrong customer. It writes a database migration that drops a column three services still read from. In each case the model produced a plausible-looking action, executed it, and moved on. And in each case somebody absorbed a real cost — a rebooking fee, a damaged relationship, an incident bridge at 2 a.m.

Here is the uncomfortable part: most AI products have no answer for who that somebody is. The question never comes up in the design review. It surfaces later, one ticket at a time, in a support queue where an agent improvised a $40 credit because the customer sounded angry and the rep had no policy to point at. Multiply that by a few thousand tickets a month and the unit economics quietly rot — not from a dramatic failure, but from a slow leak nobody scoped.

"The model made a mistake" is not a support escalation. It is a billing event. And the products that survive the agentic era will be the ones that designed for that event before the first angry ticket, not the ones that improvised refunds by vibes until the gross margin went negative.

Every agent framework now ships "long-term memory" as a headline feature, and every team adopts it as an unambiguous good. The agent remembers the user's preferences, prior decisions, project context, and the corrections it was given last week, so each session starts warmer than the last. The demo is irresistible: a user says "set up the project the way I like it" and the agent just does it. Nobody asks the obvious question, because the framing of the feature actively discourages it.

The question is: when does any of that stop being true?

A memory store is a cache. It holds facts about a world that does not hold still. The agent recorded "the user prefers Postgres" eight months ago, and the team has since migrated to a different database. The agent remembers "the user is on the growth team," and the user changed roles in March. The agent stored a tidy summarized conclusion from a conversation whose premises were corrected two messages later. And the memory layer surfaces all of it with exactly the same confident freshness as a fact written this morning. We have spent fifty years learning that a cache without an invalidation policy is a correctness bug. Then we built agent memory and shipped it without one.

Somewhere in the evolution of every AI feature, a reviewer asks a reasonable-sounding question: "Can we have the model tell us how confident it is, so we can route the low-confidence answers to a human or a fallback?" It sounds like free insurance. You add a confidence field to the output schema, the model dutifully fills it in, and now you have a dial to turn. Ship it.

That dial is not free, and worse, it is usually not wired to anything. The confidence number is a token sequence the model is happy to produce and under no obligation to mean. Teams pay real tokens and real latency to acquire it, never check whether it correlates with correctness, and then route production traffic on it as if "0.9" were a 90% reliability estimate. It is a gauge bolted to the dashboard with nothing behind the glass.

This post is about the two costs nobody priced: the per-request tax of generating the confidence field at all, and the much larger cost of trusting an uncalibrated number to make routing decisions.

Every distributed system you have ever built leans on one quiet assumption: a retry after a timeout is safe. The operation is idempotent, so if the client gives up waiting and re-sends, the worst case is duplicate work that converges to the same state. Two PUTs land the same row. Two DELETEs leave the same absence. The retry is a no-op dressed as a second attempt.

LLM calls break this assumption, and they break it silently. A retry does not re-fetch the same answer — it samples a new one. When a client times out at the network layer because the response was lost in transit, but the provider actually finished the generation, the retry produces a second, different answer. Now two distinct outputs exist for one logical request, and nothing in your stack knows which one is canonical.

This is not a rare edge. Practitioners running models behind timeouts report that 5–10% of requests hit the full timeout-plus-retry cycle even when the underlying call eventually succeeds. Every one of those is a coin flip your system was never designed to adjudicate.

Your availability dashboard says 99.95%. Your users say the answer stopped mid-sentence. Both are correct, and that is the problem.

The HTTP-era reliability stack was built on a single assumption: the status code arrives at the end of a request and summarizes its fate. A 200 means success. A 5xx means retry. The load balancer counts the ratio, the SLO dashboard aggregates it, the alerting fires on the burn rate. Every layer of that stack reads the header and trusts it.

Streaming inverts the assumption. The moment your server flushes the first token, it has already committed to a 200. Everything that goes wrong after that — a provider timeout at token 400, a content filter trip mid-paragraph, a dropped TCP connection, a malformed tool-call fragment — happens after the verdict has been rendered and cannot be retracted. The request failed. The status code says it succeeded. And nothing in your reliability tooling is built to notice the difference.

The pitch sounded responsible. "Let's not hardcode OpenAI everywhere — we'll put a thin abstraction in front, then we can swap providers if we need to." Two years later, that thin abstraction is a service with its own deploy pipeline, its own SRE on-call, an eval gate that blocks bad prompts, a semantic cache that saves seven figures a year, a retry policy with provider-specific backoffs, an observability schema every dashboard depends on, and a key vault holding the credentials for six model vendors. Every AI feature in the company terminates there.

It is also, almost by accident, the single point of failure with the worst blast radius in the stack. When the primary LLM provider goes down — and in 2025 OpenAI was tracked having 294 outage events since January, with Anthropic logging 184.5 hours of total customer impact in December alone — the gateway routes around it and most users never notice. When the gateway itself dies, every AI feature in every product simultaneously stops, the failover that was supposed to fire never gets a chance, and the postmortem opens with "the abstraction layer we built to insulate us from provider outages was the outage."

Your eval suite ran green at 2 AM PT on a quiet provider. QA smoke-tested at 11 PM the night before launch. The feature goes live, and by Tuesday at 10 AM Eastern your p95 is 40% higher than the dashboard you signed off on, your agent is dropping the last tool call in a six-step plan, and your support inbox is filling with tickets that all sound the same: "the AI was weird this morning." Nobody is wrong. The model is also not wrong. The eval set is wrong — it never saw a saturated provider, so it has no opinion on what the feature does when the queue depth triples and the deadline budget collapses.

Provider load is not a latency problem with a quality side effect. It is a distribution shift in the inputs your model and your agent loop receive, and you have built every quality signal you trust on the wrong half of that distribution. The fix is not a faster region or a better model. The fix is to stop pretending your eval harness is sampling from the same world your users are.

Every team that ships agents to production eventually wakes up to the same kind of incident. An agent enters a state it cannot exit. It re-calls the same tool with cosmetically different arguments for six hours. It oscillates between two plans whose preconditions reject each other. It retries a transient 429 every two hundred milliseconds until morning. It generates a million-token plan it never executes. By the time anyone notices, the token bill is four figures, the downstream API is rate-limited, the customer's session has timed out twelve times, and the on-call engineer is being paged by three different alerts about the same root cause.

The first fix every team reaches for is a step-count budget. Cap the agent at twenty iterations. Cap it at fifty. Pick a number and ship. The step budget makes the incident reports stop, but it does not make the underlying problem go away — and once you understand the mechanism, you can see why a step budget is the agent equivalent of a household fuse: it blows after the damage has been done, the fuse box itself is now a maintenance burden, and the next time something melts, your reflex is to swap in a higher-rated fuse rather than ask what is actually shorting.

A team ships a self-critique loop into production because the benchmark numbers looked irresistible: Self-Refine reported a 20 percent absolute improvement averaged across seven tasks, Chain-of-Verification cut hallucinations by 50 to 70 percent on QA workloads, and reflection prompts pushed math-equation accuracy up 34.7 percent in one widely-cited paper. A month later the finance review surfaces the bill. The product's per-request cost has roughly tripled, p99 latency is up by a factor of three, and the actual quality lift that survived contact with production traffic is closer to three percent than thirty. The self-critique loop is doing exactly what it advertised. The team just never priced it.

This is the self-critique tax: a reliability pattern that reads like a free quality win on a slide and reads like a structural cost increase on an invoice. The pattern itself is sound — there are real cases where generate-then-verify is the right answer. The failure mode is shipping it as a default instead of as a calibrated intervention, and discovering at the wrong time of the quarter that "the model checks its own work" was actually a procurement decision.