278 posts tagged with "reliability"

Classic chaos engineering rests on a quiet assumption: if you inject the same fault twice, you get the same failure twice. Kill the pod, watch the failover, fix the gap, kill the pod again to confirm. The entire discipline — hypothesis, blast radius, steady-state metrics — presumes a system deterministic enough that the experiment is repeatable.

Agent systems break that assumption at the root. Inject a tool timeout into an agent run and the model reroutes its plan — one time it retries, another time it substitutes a different tool, a third time it confidently fabricates the result it never fetched. Run the identical fault against the identical prompt and you get a different trajectory, because the failure path runs through a stochastic planner. The failure you saw in production last Tuesday will never happen again in exactly that shape. And that's precisely why you have to rehearse it anyway.

Ask any infrastructure team to restore the production database to yesterday at 3pm and they will quote you a runbook, an RPO, and a time estimate. Ask the same team to restore the agent to yesterday at 3pm — before it absorbed a batch of poisoned memories, before someone shipped the bad prompt revision, before the reindex that quietly broke retrieval — and you will get silence. Not because the individual pieces lack backups, but because nobody can say what "the agent at 3pm" even means.

That is the uncomfortable discovery waiting for every team running agents that learn: your database has snapshots, your code has git, and your agent — the thing your users actually interact with — has neither. Its operational state is smeared across a vector index, a pile of memory files, a prompt registry, and a set of tool configurations, each versioned independently or not at all. Restore any one of them alone and you don't get yesterday's agent back. You get an incoherent brain.

Somewhere in your inference fleet, there is probably a chip that computes wrong answers. Not crashes — wrong answers. It passed manufacturing tests, it passes health checks, and under a specific combination of instruction sequence, data values, voltage, and temperature, it returns a number that is simply incorrect. The hyperscalers have documented this at scale: roughly one device in a thousand silently corrupts data, a rate several orders of magnitude worse than the cosmic-ray bit flips we used to worry about.

For fifty years, software had an immune system against this: determinism. Same input, same output — so you could checksum, replay, and compare against golden results, and lying hardware eventually got caught. LLM inference is the first major workload where that immune system is gone. When a model gives a slightly worse answer, was it the sampler being a sampler, or a degraded GPU flipping bits in your KV cache? Nobody can tell by inspection. A flaky accelerator can quietly drag down your quality metrics for weeks while every dashboard stays green.

For fifty years, data pipelines enforced their contracts by dying. An upstream team renamed a column, the downstream parser threw, the job crashed, someone got paged at 2 a.m., and by morning the contract was either fixed or formally renegotiated. Nobody designed this as a governance mechanism. It just fell out of the fact that rigid code cannot process input it doesn't expect. The crash was the enforcement. The pager was the audit trail.

Then we put an LLM in the consumer seat, and the breakage stopped crashing.

A model reading a malformed record doesn't throw a parse error. It copes. A missing field becomes a plausible guess. A renamed field becomes a slightly wrong interpretation. A unit change — cents to dollars, UTC to local — becomes a confident answer that's off by a factor the model never mentions. The pipeline runs green end to end, the dashboards stay quiet, and the broken contract surfaces three weeks later as a diffuse quality complaint that nobody can bisect. We didn't remove the failure. We removed the signal.

Ask an infrastructure team what happens if the primary database goes down and you will get a rehearsed answer: replicas, failover runbooks, RTO and RPO numbers someone signed off on. Ask the same team what happens if the model API goes down and you will usually get a shrug and a link to the provider's status page. That asymmetry made sense in 2023, when the LLM powered an experimental sidebar. It stopped making sense the day your support flow, your search ranking, your code review bot, and your onboarding assistant all started routing through one vendor's inference endpoint.

The model API is now a tier-0 dependency for a lot of products — revenue-critical, sitting in the request path next to the database — but most disaster-recovery plans still treat it like a nice-to-have integration. The result is a familiar incident shape: the provider degrades, every AI feature in the product throws the same spinner, on-call stares at a status page they can't influence, and nobody can answer the only question that matters: what is this product supposed to do right now?

The incident channel says the new prompt is hallucinating refund amounts, so you do the obvious thing: repoint the production tag to last week's prompt version. Thirty seconds, no redeploy, textbook rollback. Except the agent gets worse. Last week's prompt references a lookup_order tool that the platform team renamed to orders.search on Tuesday. The memory store is full of preference summaries written by the new prompt's format, which the old prompt reads as user instructions. You didn't roll back the agent. You built a chimera — one-third last week, two-thirds today — and shipped it to production without ever testing that combination.

This is the failure mode nobody's runbook covers: an agent deployment is not an artifact, it's a triple — prompt version, tool contract, accumulated memory state. Rolling back one leg while the other two advance doesn't restore a previous state. It creates a new state that has never existed before, never passed an eval, and belongs to no team's on-call rotation.

Pull up your request logs from the last time you hit sustained 429s. You will probably find something odd: the errors don't arrive as a steady stream. They arrive in waves — a burst of 429s, a quiet gap, a bigger burst, another gap. The provider's quota didn't change between waves. Your traffic didn't spike. What you are looking at is your own retry logic, synchronized against itself. Every client that failed at second zero computed the same backoff delay, slept the same duration, and woke up at the same instant to fail together again.

This is the thundering herd, and the punchline is that the standard fix — exponential backoff — does not prevent it. Deterministic exponential backoff organizes the herd. It takes a crowd of clients that failed at roughly the same moment and marches them forward in lockstep: everyone retries at 1 second, then everyone at 2, then everyone at 4. The load spikes get farther apart, but each spike is just as tall as the first. If the spike is what triggered your rate limit, you have built a metronome that re-triggers it forever.

Your agent died at step seven of a twelve-step task. The framework caught the exception, waited with exponential backoff, and retried. It retried the step — with the same context window that had accumulated three failed tool calls, a half-parsed error message, and a plan the model had already abandoned. The retry failed too, of course, because a retry is a bet that the world changed, and nothing about that agent's world had changed. What needed to change was the agent's state — and no retry policy in any agent framework makes that decision.

Erlang's OTP libraries codified this exact decision thirty years ago, for telephone switches that had to run for decades. The insight behind supervisor trees was never "restart things when they crash." It was that how to recover is a separate concern from doing the work, owned by a separate process, arranged in a hierarchy where each level knows a little more about what recovery means. Most agent frameworks today bolt retries onto individual calls, which is like putting a try/catch around every line of a telephone switch. What they need is the hierarchy.

Here is a postmortem pattern that is becoming a genre. The primary model was healthy all night. Latency was flat, token throughput normal, provider status page green. And yet every user request failed for forty minutes — because the safety classifier sitting in front of the model timed out, and the middleware wrapped that timeout in a generic exception, and the exception handler returned a refusal. Your model didn't go down. Your gate went down, and the gate was wired to fail closed by an engineer who never thought of it as a decision.

The uncomfortable truth is that most teams run a second machine-learning system in production without admitting it. The moderation classifier, the jailbreak detector, the PII scrubber, the topical filter — each one is a model, with its own latency distribution, its own error rates, its own training-data assumptions quietly rotting under drift, and its own failure modes. But because it's called a "guardrail," it gets treated like a config file: set once, never monitored, absent from the dashboard, missing from the on-call runbook. You would never ship your primary model without an SLO. Most teams ship their guardrail without even a health check.

Rename a database column and watch what happens. The compiler catches every query builder. The ORM migration catches the model class. The type checker catches the API serializer. Integration tests catch the two services that read the field over the wire. Every consumer of that column gets flagged — except one. Your system prompt, which contains a carefully formatted description of the table "so the model understands the data," keeps confidently describing a column that no longer exists. No compiler error. No failing test. No deprecation warning. Just a model that starts generating queries against a schema from three sprints ago, and a dashboard that slowly fills with malformed SQL.

Prompts have foreign keys. They reference database schemas, tool signatures, enum values, API shapes, and few-shot examples copied from production data — and none of those references participate in refactoring. Every fact you paste into a prompt is a join against a table that your tooling doesn't know exists. When the referenced artifact changes, nothing cascades. The prompt just rots in place.

A refund agent processes a customer request for $42. The tool call it emits is refund(amount: 4200) — wait, is that right? If the backend stores money in cents, it's exactly right. If the backend stores dollars, the customer just got a $4,200 refund. Both calls are syntactically perfect. Both pass JSON Schema validation in microseconds. The schema says amount is a number, and 4200 is unimpeachably a number.

This is the bug class that unit tests, schema validators, and most agent evals all sail past: unit confusion at the tool-call boundary. The model pattern-matches magnitudes from its training data — money in dollars, time in seconds, weight in whatever the surrounding prose implied — while your tool expects cents, milliseconds, and kilograms. Nothing in the type system objects. The invoice is just off by 100x.

We spent two decades teaching services to retry safely. The playbook is well worn: a client generates a unique idempotency key, attaches it to the request, and the server records the key alongside the result inside the same transaction that performs the work. A dropped connection, a timeout, a 500 — the client retries with the same key, the server recognizes it, and returns the recorded result instead of charging the card twice. Stripe shipped this pattern years ago and it became table stakes for any API that touches money.

That entire design rests on one assumption nobody wrote down: the caller repeats its request byte-for-byte. The retry carries the same key because the retry is the same code path re-executing with the same variables. Break that assumption and the whole scheme quietly stops working.