43 posts tagged with "incident-response"

The incident bridge is quiet in a way that means everyone is stuck. A support ticket says the agent told a customer their refund was approved when it wasn't. You have the full trace: the prompt, the retrieved account record, the tool calls, the model's reasoning, the final message. You replay it. The agent does the right thing. You replay it again. Right again. Nine times out of ten, the trace that produced the incident produces a correct answer instead. Someone on the call finally asks the question that the retro template can't handle: so what was the root cause?

There isn't one. Not in the sense the template means. The five-whys chain runs "the agent told the customer the wrong thing" → "because the model generated an approval" → "because it sampled a token sequence that asserted approval" → "because... that's what the probability distribution allowed." The last why bottoms out in a shrug. "The model sampled a bad token" is technically true and operationally useless. It names no fix, assigns no owner, and closes no gap. You can write it in the report, but everyone reading it knows you've documented a coincidence, not a cause.

Classic chaos engineering rests on a quiet assumption: if you inject the same fault twice, you get the same failure twice. Kill the pod, watch the failover, fix the gap, kill the pod again to confirm. The entire discipline — hypothesis, blast radius, steady-state metrics — presumes a system deterministic enough that the experiment is repeatable.

Agent systems break that assumption at the root. Inject a tool timeout into an agent run and the model reroutes its plan — one time it retries, another time it substitutes a different tool, a third time it confidently fabricates the result it never fetched. Run the identical fault against the identical prompt and you get a different trajectory, because the failure path runs through a stochastic planner. The failure you saw in production last Tuesday will never happen again in exactly that shape. And that's precisely why you have to rehearse it anyway.

The first fifteen minutes of almost every incident are mechanical. Pull the four graphs everyone always pulls. Diff the deploys against the incident start time. Check which feature flags flipped today. Search the runbook wiki for the error string. None of this requires judgment — it requires being awake, and at 3 AM your on-call engineer is spending those minutes finding their laptop, joining the bridge, and remembering which dashboard is the real one. An agent can have all of it done before the human's screen unlocks.

Yet in most organizations, the conversation about AI in incident response dies at a single anecdote: "we heard about a team whose auto-remediation script took down production." So the whole idea gets banned — not scoped, not staged, banned. This is a category error. The horror stories are about the top rung of an autonomy ladder, and teams respond by refusing to step on the bottom rung, where the agent has no write access to anything and the worst it can do is post a wrong paragraph in Slack.

The pager goes off at 2:14 a.m. A checkout service is throwing 500s, revenue is bleeding, and you are the on-call engineer. You pull up the failing module and start reading. The code is clean — well-named functions, sensible structure, even a few helpful comments. And you have no idea what it does. You didn't write it. Nobody on your team really wrote it. An agent generated it four months ago, it passed review, the tests went green, and it has been running in production ever since. Now it's on fire, and the person who is supposed to fix it is meeting it for the first time.

This is comprehension debt: the widening gap between the amount of code your organization runs and the amount any human actually understands. It doesn't show up on a dashboard. It accrues silently while everything looks healthy, and it comes due at the worst possible moment — during an incident, when the cost of not understanding your own system is measured in downtime.

The deprecation notice arrived on a Tuesday with a sunset date six months out. Your platform team logged it in the dependency tracker with a "Q3 cutover" label and a yellow severity. It joined two other migrations already in the queue. Three weeks later, the provider amended the date inside the same URL, no diff, no inbox notification, just a quietly updated paragraph that pulled the sunset sixty days earlier into the middle of your code freeze.

The lifecycle page you treated as a planning document was always a contract clock. The only thing that changed is which team's calendar it controlled — and the team that owns it is not yours.

The page fires at 02:17 local time. The on-call SRE pulls up the agent runbook on their phone and reads step one: "check the agent's tool-call traces for anomalous tool usage." They open the link. They hit an SSO prompt for a workspace they do not belong to. Step two says inspect the prompt-construction logs; same wall. Step three says roll back to the previous prompt version, but the deploy permission is scoped to a team they are not on. By the time they figure out which Slack channel to escalate to and wake up the AI team's product manager because she is the only person they can find at 02:17, ninety minutes have passed and the customer-visible regression is still serving wrong answers.

The post-mortem will identify the access gap as the proximate cause. The deeper discomfort is that the runbook reads fine in daylight and runs blocked at night, because the person who wrote it has access the person who executes it does not.

The runbook said "disable the agent." The on-call followed it. Forty-three minutes later, when the kill switch finally propagated through the config service, the agent had already filed 1,200 incorrect tickets, called the billing API 8,000 times, and sent emails to customers who hadn't signed up for any of it. The runbook was correct. The runbook was also useless, because nobody had ever measured how long "disable the agent" actually takes when an agent is producing damage by the second.

Most AI features ship with a kill switch the same way most buildings ship with a fire extinguisher: someone signed off that it exists, nobody timed how long it takes to reach. The compliance review asks "is there a kill switch?" and the answer is yes. The incident asks "how fast does it stop the bleeding?" and the answer is whatever the underlying plumbing happens to take — a number nobody on the team has ever measured against the rate at which the feature is doing harm.

The mismatch is the whole problem. A feature whose containment time is longer than its blast time has shipped containment theater.

The page fired at 02:14. The runbook said "page the engineer." The engineer's name resolved to an on-call rotation. The rotation pointed at a Slack channel that the team had wired up six months ago as a unified triage surface. The first message in the channel was the alert. The second message, posted nineteen seconds later, was a calm three-sentence summary: the alerting service, the failing dependency, the last deploy. It was well-written. It ended with "Acknowledged."

The incident commander, watching from her phone in bed, read "Acknowledged" and went back to sleep. Nobody had acknowledged. The agent subscribed to that channel as a first-line triage helper had restated the alert back to the room and signed off with the verb the channel's other readers used to mean "I have the context to act on this." The incident ran unowned for forty-one minutes until a customer ticket woke a different engineer through a different surface.

The first time an agent caused a real outage on my team, the postmortem author opened the template, scrolled past the timeline, stared at the "Root Cause" field for a long minute, and typed: "The runbook for queue-stuck recovery was incorrect." The runbook was fine. The agent had read the runbook, decided the queue's symptoms matched a different scenario, and run a recovery script for that other scenario instead. The action items that came out of that document — "tighten the runbook wording," "add a confirmation prompt to the recovery script" — were entirely useless against the actual failure mode, which was that an inferential system had inferred wrong and there was no field in the template that knew how to say so.

I've watched this exact failure repeat across teams since. The template is calibrated for deterministic systems. Code did the wrong thing, so you fix the code. Config was misset, so you fix the config. The schema of the postmortem document is the schema of the team's theory of failure, and when that theory cannot represent "the agent's plan was wrong," the document flattens the actual failure into the closest thing the template can represent — usually a documentation gap or a missing guardrail — and the action items chase a deterministic fix for a probabilistic failure. The same incident class then recurs, and the team writes it up the same way the next time.

The launch flag worked perfectly. You shipped the AI summarizer behind it, ramped 1% to 10% to 50% to 100% over two weeks, watched the dashboards, saw nothing on fire, and at the end of the quarter the platform team's flag-hygiene bot opened a PR to delete the now-redundant gate. You approved it. The PR merged with the rest of the expired-flag cleanup, and the codebase got 200 lines lighter. Six weeks later at 2am, the provider rolls a fresh model snapshot, your summarizer starts confidently fabricating clauses into legal documents, and your on-call engineer discovers there is no fast lever to turn it off — only a deploy.

The flag did its job. The flag was the wrong artifact to keep. A launch flag answers "should this new code path be reachable?" and once everyone agrees yes, deleting it is the correct hygiene move. A kill switch answers "is the upstream model behaving today?" — and that question never expires, because the upstream model never stops changing. Cleaning them up together is the same category error as treating a smoke detector like a construction permit: the permit gets archived once the building is up, but the detector stays wired forever because the thing it watches for can still happen.

The incident review went smoothly right up until the question that nobody could answer. Structured-output errors had spiked at 2:14pm, a revenue workflow had stalled for ninety minutes, and the timeline reconstructed cleanly: a system prompt had been edited three weeks earlier, and a few extra words about "conversational tone" had quietly pushed the model off its JSON contract under certain inputs. The fix was a one-line revert. The hard part came next. Someone asked who had made the change, and who had reviewed it, and which team owned that prompt going forward. The room went quiet. There was no pull request. There was no reviewer. The edit had been made in a vendor dashboard at 11pm by someone who no longer remembered doing it.

That silence is the actual incident. The JSON contract breaking was a symptom. The root cause was that the single highest-leverage piece of behavior in the system had no owner, no change history, and no path through the process that governs every other production change. The model didn't fail. The model did exactly what it was told. The failure was that the telling had escaped change management entirely.

This is one of the most common production AI incidents right now, and it almost never gets named correctly. The postmortem writes "prompt regression" in the root cause field and moves on. But "prompt regression" describes the code. The real root cause is an org chart with a hole in it.

The on-call rotation was built around a promise: failures reproduce. An alert fires, you re-run the request, you watch the bug happen, you find the bad commit, you roll back the deploy. Every part of that loop assumes determinism. The same input produces the same output, and the output is either right or wrong in a way you can stare at.

An agent fleet quietly breaks every link in that chain. The failure happened once, at a sampling temperature you can't replay, on a context window that has since been garbage-collected. There is no bad commit, because the code never changed — the model did, or the retrieved documents did, or the user phrased the request in a way nobody anticipated. You roll back the deploy and the deploy was never the problem.

So the page goes out, an engineer picks it up, and they discover the most uncomfortable fact about operating agents in production: they have been handed a system they cannot single-step, and the runbook in front of them was written for a different kind of machine.