Let the Agent Answer the Page First: A Trust Ladder for AI Incident Response

The first fifteen minutes of almost every incident are mechanical. Pull the four graphs everyone always pulls. Diff the deploys against the incident start time. Check which feature flags flipped today. Search the runbook wiki for the error string. None of this requires judgment — it requires being awake, and at 3 AM your on-call engineer is spending those minutes finding their laptop, joining the bridge, and remembering which dashboard is the real one. An agent can have all of it done before the human's screen unlocks.

Yet in most organizations, the conversation about AI in incident response dies at a single anecdote: "we heard about a team whose auto-remediation script took down production." So the whole idea gets banned — not scoped, not staged, banned. This is a category error. The horror stories are about the top rung of an autonomy ladder, and teams respond by refusing to step on the bottom rung, where the agent has no write access to anything and the worst it can do is post a wrong paragraph in Slack.

The fix is not more courage. It is a ladder with explicit rungs, explicit evidence requirements for promotion, and — this is the part everyone skips — explicit demotion criteria that stop one bad episode from becoming a permanent ban.

The Asymmetry Nobody Prices: Triage Is Free, Remediation Is Not

Incident work splits into two kinds of actions with wildly different risk profiles, and lumping them together is what kills the ladder before it starts.

Read-only investigation — correlating latency spikes with recent commits, walking the service dependency graph, pulling the last deploy's diff — has a failure cost measured in attention. If the agent's summary is wrong, an engineer reads a misleading paragraph and discards it. Modern incident tooling already works this way: investigation agents activate the moment an alert fires and run parallel hypothesis checks before the team has finished reading the notification, citing the specific logs and code changes behind each hypothesis.

Write actions — restarting services, rolling back deploys, scaling clusters — have a failure cost measured in blast radius. A wrong restart during a partial outage can turn it into a full one; Google's SRE book devotes a whole chapter to how well-intentioned mitigation traffic and restarts amplify cascading failures. The S3 outage of 2017 started with one overly broad removal command; the lesson Amazon drew was not "never automate" but "bound what any single command can do."

These two categories deserve different trust thresholds, different rollout speeds, and different failure budgets. The ladder makes that explicit.

The Four Rungs

Rung 1 — Narrator. The agent has read-only access to observability, deploy history, feature flags, and the runbook corpus. When a page fires, it posts a triage summary to the incident channel: what changed recently, which graphs look anomalous, which runbook sections match, what it believes the leading hypothesis is — with links to the evidence. Humans execute everything.

The agent cannot touch production, so the only risk at this rung is a wrong narrative. That risk brings its own discipline: every claim must carry a citation an engineer can verify in under thirty seconds. A hypothesis without linked evidence is noise wearing a confidence score.

Rung 2 — Proposer. The agent drafts specific actions — "roll back deploy 4f2a, it correlates with the error onset" — as approval cards in the incident channel. A human clicks approve; the agent executes and reports the result. The judgment stays human; the typing moves to the machine. This rung matters more than it looks: it converts the agent's proposals into a labeled dataset. Every approval and rejection is a training signal and, more importantly, an audit trail you will use to argue for (or against) promotion.

Rung 3 — Gated executor. For a small, enumerated set of runbooks — clear a full disk on a stateless node, recycle a wedged worker pool, fail over to the warm standby — the agent executes without waiting for approval, but inside hard bounds: rate limits, blast-radius caps ("no more than one node at a time"), pre-flight state checks, and post-action verification with automatic escalation if the metric doesn't recover. The runbook list is an allowlist, reviewed like code, and every entry got there by demonstrating a track record at rung 2.

Rung 4 — Bounded auto-remediator. The agent handles a class of incidents end to end — detection through verified recovery — and pages a human only when its confidence drops or its bounds are hit. Most teams should not be here for most incident classes, and that is fine.

Loading…