52 posts tagged with "sre"

Your agent has a latency SLO. Someone put it in a doc: "responses under 8 seconds, p95." What nobody did was decide how those 8 seconds get spent. There is no line item for the retrieval call, no line item for the planning step, no line item for the third tool the model decided to invoke because it felt uncertain. The budget exists as a single number at the boundary and as nothing at all inside. So when a five-hop chain blows past 8 seconds, the on-call engineer stares at a trace and cannot answer the only question that matters: which hop did it?

This is the difference between a service that has a latency budget and a service that has a latency hope. A budget is allocated per component and enforced. A hope is measured at the door and prayed over. Most agent systems ship with a hope, because the hop structure is dynamic — the model decides how many tool calls to make — and it feels impossible to budget something you don't control. It isn't. You budget it exactly because you don't control it.

Classic chaos engineering rests on a quiet assumption: if you inject the same fault twice, you get the same failure twice. Kill the pod, watch the failover, fix the gap, kill the pod again to confirm. The entire discipline — hypothesis, blast radius, steady-state metrics — presumes a system deterministic enough that the experiment is repeatable.

Agent systems break that assumption at the root. Inject a tool timeout into an agent run and the model reroutes its plan — one time it retries, another time it substitutes a different tool, a third time it confidently fabricates the result it never fetched. Run the identical fault against the identical prompt and you get a different trajectory, because the failure path runs through a stochastic planner. The failure you saw in production last Tuesday will never happen again in exactly that shape. And that's precisely why you have to rehearse it anyway.

The first fifteen minutes of almost every incident are mechanical. Pull the four graphs everyone always pulls. Diff the deploys against the incident start time. Check which feature flags flipped today. Search the runbook wiki for the error string. None of this requires judgment — it requires being awake, and at 3 AM your on-call engineer is spending those minutes finding their laptop, joining the bridge, and remembering which dashboard is the real one. An agent can have all of it done before the human's screen unlocks.

Yet in most organizations, the conversation about AI in incident response dies at a single anecdote: "we heard about a team whose auto-remediation script took down production." So the whole idea gets banned — not scoped, not staged, banned. This is a category error. The horror stories are about the top rung of an autonomy ladder, and teams respond by refusing to step on the bottom rung, where the agent has no write access to anything and the worst it can do is post a wrong paragraph in Slack.

Every latency playbook you own was written for systems where the work per request was roughly constant. A database lookup takes what it takes. An image resize scales with pixel count, which you know in advance. Even classic LLM completions had a knowable cost envelope: input tokens in, bounded output tokens out. Reasoning models quietly deleted that assumption. When the model decides at runtime how long to think — and it decides based on how hard the problem turns out to be — response time stops being a property of your infrastructure and becomes a property of the question.

The consequence shows up first in your percentiles. Teams running reasoning models in production report p99 latencies spiking three to five times above p50, not because a host got slow or a cache went cold, but because one request in a hundred happened to be genuinely difficult. Your autoscaler, your timeout policy, and your SLO dashboard were all tuned for a world where that spread meant something was broken. Now it means the system is working as designed — and every tool you have for managing the tail is pointed at the wrong cause.

The pager goes off at 2:14 a.m. A checkout service is throwing 500s, revenue is bleeding, and you are the on-call engineer. You pull up the failing module and start reading. The code is clean — well-named functions, sensible structure, even a few helpful comments. And you have no idea what it does. You didn't write it. Nobody on your team really wrote it. An agent generated it four months ago, it passed review, the tests went green, and it has been running in production ever since. Now it's on fire, and the person who is supposed to fix it is meeting it for the first time.

This is comprehension debt: the widening gap between the amount of code your organization runs and the amount any human actually understands. It doesn't show up on a dashboard. It accrues silently while everything looks healthy, and it comes due at the worst possible moment — during an incident, when the cost of not understanding your own system is measured in downtime.

A real regression burned for two days in production. The page had fired. It had fired correctly, at the right threshold, with the right severity. Three weeks earlier the on-call rotation had added a silence rule for that alert family because every page in that family had so far resolved with the same comment: "nothing to do, investigating." The post-mortem could not honestly call the silence a mistake. It was a rational adaptation to a stream of pages the rotation had no playbook for. The regression that mattered shipped against a muted channel because the team's monitoring stack was producing signals it could not act on, and the team's response to that was the only one available: stop listening.

This is not an alerting bug. It is a structural property of how AI features get instrumented when teams reach for the playbook they already know. Latency, error rate, refusal rate, output schema conformance, judge-eval drift — each one is a defensible metric. Each one fires with the same diffuse "model behavior changed" wording. None of them tells the on-call engineer what to do, because no one has written the runbook that maps each signal to an action, because most of the time the signal does not map to an action. The rotation absorbs the noise until the noise is louder than the signal, then it routes around the channel that produces it.

The weekly review slide said 99.9%. The invoice said the bill had tripled. The two numbers had been on adjacent dashboards for months, and nobody had noticed that they were measuring different worlds. The reliability number was post-retry — every call that eventually returned a 200 counted as a success — and the cost number was every attempt the client made, billed by the token. Between them sat a generous five-attempt retry loop and a provider whose tail latency had been quietly degrading. The first time anyone looked at both numbers together was during an outage, when the cost-anomaly alert fired before the availability alert did.

That is the whole pattern. A retry budget that looks like a reliability mechanism is also a cost-quality knob, and the team that watches only one side of it is paying for an availability number the invoice will eventually correct.

The page fires at 02:17 local time. The on-call SRE pulls up the agent runbook on their phone and reads step one: "check the agent's tool-call traces for anomalous tool usage." They open the link. They hit an SSO prompt for a workspace they do not belong to. Step two says inspect the prompt-construction logs; same wall. Step three says roll back to the previous prompt version, but the deploy permission is scoped to a team they are not on. By the time they figure out which Slack channel to escalate to and wake up the AI team's product manager because she is the only person they can find at 02:17, ninety minutes have passed and the customer-visible regression is still serving wrong answers.

The post-mortem will identify the access gap as the proximate cause. The deeper discomfort is that the runbook reads fine in daylight and runs blocked at night, because the person who wrote it has access the person who executes it does not.

The AI platform team has four engineers. The internal agent they shipped seven months ago is now answering questions for 200 employees a day. For the first month the founding engineer answered every Slack ping personally — Tuesday at 11pm, Sunday morning, the night of the company offsite. Then she got promoted to staff engineer for the impact she had on adoption, and three weeks later she stopped checking the channel after 6pm because that is what staff engineers do. The on-call rotation that was supposed to replace her was never formalized, because the operating model was always going to be figured out "after the pilot."

The day the agent silently degrades for a quarter of users — a retrieval index that quietly fell behind, or a model version flip that shifted refusal behavior, or a tool whose schema rotated and is now returning empty arrays — the complaints do not land on the platform team's pager. They land in the help desk queue, staffed by people who do not have access to the agent's traces, do not know what a system prompt is, and have been told by IT that the agent is "owned by the AI team." Sixteen hours pass between the first user complaint and the first engineer who looks at a trace. Nobody on the platform team is asleep at the wheel; there is no wheel.

Your monthly token quota resets at 00:00 UTC. Your largest customer is in Tokyo and hits peak load at 21:00 UTC — 6:00 AM their next morning. By the time the reset arrives, the Tokyo workday has already chewed through the last six hours of the cycle on quota-exhaustion fallback. The 429s look "occasional" because the UTC calendar axis on your dashboard hides the daily reset boundary inside an ordinary timestamp.

This is not a rate limit bug. It is a calendar bug. The provider chose a reset clock for their bookkeeping convenience, and the geography of your traffic decided which customers got the empty end of the cycle. The team that priced the quota as a uniform resource is rationing it on a calendar the user never sees.

An agent ran. The plan-step crashed. The tool-call step retried twice with a 500, then succeeded on the fourth attempt. The user got their answer.

How many events was that? Ask product, and it's one — the user got a working result, so the funnel counts a conversion. Ask SRE, and it's three failures plus one success, a 75% error rate on the underlying step. Ask finance, and it's four billable inferences, two retried tool calls, and roughly four times the unit cost product is forecasting against. Each team's dashboard is correct. They are also irreconcilable, and the moment someone tries to reconcile them — usually during an incident review — they will discover the team has been operating against three contradictory pictures of reliability for months.

Streaming was a product decision. Somebody on the design team watched a competitor's chat UI tick out tokens like a typewriter, watched a user's shoulders relax when the first character appeared two hundred milliseconds in instead of after a four-second blank stare, and the decision was made: we stream. The pull request changed three files in the API gateway. The model output now flushes incrementally over Server-Sent Events. The launch went out on a Tuesday and the satisfaction score moved up by a measurable amount on a Wednesday. Nobody opened a ticket against infrastructure.

A month later the on-call engineer is staring at three dashboards that no longer agree with each other. The autoscaler is provisioning twice as many pods as the CPU graphs say it should need. The p99 latency dashboard is broken — not malfunctioning, but uninterpretable, because the histogram buckets stop at five seconds and most spans now live in the overflow. The capacity model that priced the previous quarter's bill said the service could handle twelve hundred requests per second per node. The graph in front of the on-call says it is handling four hundred and falling over.