278 posts tagged with "reliability"

A long-running agent generates a plan at step 3. The plan reads something like: "if the order returned by get_order in step 1 has status shipped, send the customer a tracking email; otherwise open a refund ticket." The agent confidently picks the email branch. The customer never received a tracking number, because the order was actually in pending. You go to the trace expecting to find a hallucination. What you find is worse: the step-1 tool result is no longer in context. The pruner evicted it between step 2 and step 3 — it ranked low on recency and there was a 12KB transcript to make room for. The plan still ran. The branch was still chosen. The decision now points at evidence that does not exist.

This is not a model failure in the usual sense. The model produced a syntactically valid plan, executed it in order, and made a branch decision. The branch was made against a fact that used to be in context and is not anymore. The chain of thought encoded the condition (if status == "shipped"); the actual status got dropped on the way to the step that needed it. The plan looks deterministic, but it has been quietly cut loose from its evidence.

You added the search tool, then the calendar tool, then the CRM tool, then four database wrappers and a calculator. The dashboard moved the way you wanted: task-completion ticked up, latency held, the "I don't know" rate dropped from 14% to 4%. Looks like a capability win. It is not. The planner did not learn more; it learned less abstention. Every question now looks answerable because there is always some tool that pattern-matches the query well enough to call. The 10 percentage points of "I don't know" you removed did not turn into correct answers — they turned into confident wrong ones, distributed across the long tail where nobody is grading carefully.

This is the false-competence trap of tool surface expansion. It is the most common way a team ships a regression while celebrating an improvement. The eval rubric measures whether the agent attempted the task and produced a plausible-shaped answer; it does not measure whether the agent should have refused. Abstention is not free, but it is the cheapest correct behavior available, and you stop being able to see it the moment your tool palette gets large enough that something always fires.

The cluster did exactly what we told it to. Traffic dropped to zero for forty-five seconds, the queue-depth metric flatlined, KEDA flipped the replica count from one to zero, and the node autoscaler reclaimed the H100 pod ninety seconds later. The graph looked clean. The Slack channel was quiet. The cost dashboard ticked down half a cent.

An hour and twelve minutes later, a customer support ticket arrived: a long-running document-analysis job — a 180k-token reasoning task that was budgeted for twenty-eight minutes of decode — had vanished. No error in their client SDK. No exception in our application logs. Only a single 499 line buried in the gateway access log, timestamped roughly when the scheduler had decided the pod was idle and reaped it.

A single power user burns through your monthly token budget by 9am on day three. The kill-switch fires correctly — the gateway returns 429, the model calls stop, the bill flatlines. Meanwhile the agent has already booked the flight, sent the email confirmation, and closed the support ticket as resolved. The dashboard says "spend halted." The user says "why did you charge me for a trip I never asked for." Both are right. The budget cap stopped the model from thinking. It did not stop the world from changing.

This is the failure mode that almost every agent budget guardrail ships with: the cap is a signal in the spend plane, but the damage lives in the action plane, and the two planes were wired up with no shared transaction boundary. Telling the model to stop is not the same as telling the world to undo what the model just did.

The pager went off at 14:02 UTC. Not for latency, not for errors — for spend. The cost dashboard showed a vertical line: three minutes of input-token billing at roughly nine times the trailing hourly average, then back to normal. No regression had shipped. No tenant had onboarded. Traffic was flat to the minute. The only thing that changed is that a single prompt prefix — the 14K-token system message that every agent in the fleet shared — had quietly expired on the provider side, and a thousand workers had all decided, within the same 200ms window, that they were the ones who needed to write it back.

This is a cache stampede. It is the same bug operators have been writing post-mortems about since memcached shipped in 2003. What is new in 2026 is that the cache it stampedes is no longer yours. It lives inside your model provider, you cannot inspect its state, and every miss costs real money instead of a few extra database queries. The synchronization bug that database engineers learned to jitter away two decades ago has quietly reappeared on a bill line item nobody thought to defend.

A user asked our support agent: "Why was invoice INV-2025-08-44719 charged twice on April 3rd?" Forty-five minutes and eighteen tool calls later, the agent confidently reported back: there was no evidence of any duplicate billing on the account that quarter. The user, understandably, escalated. When we replayed the trace, the answer became obvious. The agent had compacted its conversation at turn nine. The summary said the user was "asking about a duplicate charge in early April." It did not contain the string "INV-2025-08-44719." Every subsequent tool call — the ledger lookup, the chargeback API query, the audit log scan — was issued against a paraphrased intent, not the literal invoice number the user typed.

The bug was not in the tools. It was not in the model's reasoning. It was that our context manager had a contract with every downstream component, and nobody had written it down. The contract said: "I will preserve meaning." The components needed: "I will preserve strings."

The CI test was a single assertion: same model, same temperature, same prompt, same seed, same output string. It passed on every developer's laptop, passed on the first hundred CI runs, and then flaked once every fifty runs for three weeks before anyone admitted the pattern was real. The first hypothesis was the obvious one — a non-deterministic dependency somewhere in the test harness — and three days of investigation found nothing. The actual cause was sitting in a footnote on the provider's API reference: "seed provides best-effort determinism." The team had read the parameter name and assumed a contract. The provider had documented a hint.

This is a specific failure mode of hosted inference that catches teams who design test infrastructure around a single mental model: the model is a pure function of its inputs, and the seed is what makes the function reproducible. Both halves of that model are wrong in production, and the gap between the API surface and the underlying physics is wide enough that teams build entire eval and regression-test stacks on top of an assumption their provider explicitly disclaimed.

The escalation tool was the safety net. The agent's confidence dropped below threshold, it called escalate_to_human, and the request slid into a ticket queue with a polite "a specialist will follow up shortly" reply to the user. Engineering closed the loop on the launch checklist. The on-call calendar listed humans on the receiving end.

Six months later, an audit walked the path. The escalation tool opened a Zendesk ticket. The Zendesk queue was triaged by a triage agent the support team had stood up to keep response times within SLA. The triage agent, finding no policy match it could resolve directly, called its own delegate_to_specialist tool — which routed the case to a specialist agent. The specialist agent, when uncertain, called escalate_to_human. The trace was a closed circuit. No human had touched any of the escalations the audit sampled. The human-in-the-loop the launch doc described did not exist.

The escalation interface had not failed. It had been honored at every hop. What failed was the assumption that the receiving system was a person.

The agent calls fetch_order with order_id: "ORD-739241". The schema accepts it — three letters, a dash, six digits, matches the pattern exactly. The tool returns 404. The agent hedges, generates "ORD-739242", calls again, gets another 404, generates "ORD-739243". Your dashboard records three successful tool invocations and three clean schema validations. The customer waits. Somewhere in the trace, every layer of your safety stack is reporting green while the model invents identifiers at full speed.

The team's belief is that the schema caught it. The schema caught what it could catch: shape. It checked that the argument was a string, that it matched a regex, that the required field was present. The schema cannot check that ORD-739241 corresponds to a real order in your database, because the schema does not know your database exists. That gap — between syntactic plausibility and semantic correctness — is where most production tool-calling bugs live, and the failure is so quiet that the only signal is a customer's confusion.

The runbook said "disable the agent." The on-call followed it. Forty-three minutes later, when the kill switch finally propagated through the config service, the agent had already filed 1,200 incorrect tickets, called the billing API 8,000 times, and sent emails to customers who hadn't signed up for any of it. The runbook was correct. The runbook was also useless, because nobody had ever measured how long "disable the agent" actually takes when an agent is producing damage by the second.

Most AI features ship with a kill switch the same way most buildings ship with a fire extinguisher: someone signed off that it exists, nobody timed how long it takes to reach. The compliance review asks "is there a kill switch?" and the answer is yes. The incident asks "how fast does it stop the bleeding?" and the answer is whatever the underlying plumbing happens to take — a number nobody on the team has ever measured against the rate at which the feature is doing harm.

The mismatch is the whole problem. A feature whose containment time is longer than its blast time has shipped containment theater.

The AI platform team has four engineers. The internal agent they shipped seven months ago is now answering questions for 200 employees a day. For the first month the founding engineer answered every Slack ping personally — Tuesday at 11pm, Sunday morning, the night of the company offsite. Then she got promoted to staff engineer for the impact she had on adoption, and three weeks later she stopped checking the channel after 6pm because that is what staff engineers do. The on-call rotation that was supposed to replace her was never formalized, because the operating model was always going to be figured out "after the pilot."

The day the agent silently degrades for a quarter of users — a retrieval index that quietly fell behind, or a model version flip that shifted refusal behavior, or a tool whose schema rotated and is now returning empty arrays — the complaints do not land on the platform team's pager. They land in the help desk queue, staffed by people who do not have access to the agent's traces, do not know what a system prompt is, and have been told by IT that the agent is "owned by the AI team." Sixteen hours pass between the first user complaint and the first engineer who looks at a trace. Nobody on the platform team is asleep at the wheel; there is no wheel.

A user is twelve turns into a careful conversation with your assistant about prescribing patterns for a controlled substance. The model has been measured, asking clarifying questions, citing guidance, declining to extrapolate beyond the literature. On turn thirteen, the user asks a follow-up that should land the same way the prior twelve did. Instead, they get a flat refusal: "I can't help with that." The conversation is over. They write to support furious — they were not asking anything different, the assistant was just helping them, what changed.

Your logs explain what changed. Halfway through turn thirteen, your primary provider returned a 503 in the middle of the stream. Your gateway, doing exactly what it was configured to do, failed over to the secondary provider for the remainder of the request. The secondary provider's refusal threshold for that class of query is calibrated more conservatively than the primary's. The user did not ask anything different — they asked the same question to a different model under the same brand, and the new model said no.