AI Ops Is Not Platform Engineering: How Running LLM Services Breaks Your SRE Playbook

Your SRE team is excellent at running microservices. They've mastered blue-green deployments, canary rollouts, distributed tracing, SLO burn-rate alerts, and postmortem culture. Then someone ships an LLM-powered feature, and within a week an incident happens that none of those practices were designed to handle: the model starts generating plausible-sounding but structurally wrong outputs, no error is logged, no health check fails, and users have been silently getting garbage for four hours before anyone noticed.

This isn't a skills gap. It's an architectural gap. Running LLM services is a distinct operational discipline from running microservices, and the practices that don't transfer will burn your team if you don't identify them explicitly.

The New Deployment Unit: Prompt Plus Model, Not Code

In a microservices world, the atomic unit of deployment is a container image. The image encodes all behavior, it's versioned by a digest, and swapping versions is mechanical. Behavior change implies a code change, which implies a reviewed PR, which implies a testable diff.

LLM services break this model in three ways.

Prompts are behavior that lives outside your code. A one-word change to a system prompt can flip structured JSON output into prose, weaken a safety filter, or cause a model to misinterpret a downstream API response. Prompts frequently change faster than application code—sometimes multiple times per day during active development. But they often receive none of the testing discipline applied to code: no diff review, no integration test suite, no canary validation. The result is that prompt changes are the dominant driver of production incidents in LLM applications, yet they often bypass the controls that would catch equivalent code changes.

Model updates come from outside your deploy pipeline. When your cloud provider silently upgrades the model endpoint you're calling, your application code hasn't changed. Your container didn't redeploy. Your CI didn't run. But behavior may have shifted—output verbosity, formatting conventions, refusal thresholds, reasoning patterns. Research has documented accuracy variations of up to 15% across identical runs on the same model, even at temperature zero. When a provider rotates model weights, you may not know until downstream parsing failures surface.

Configuration is a third behavioral dimension. Temperature, max tokens, top-p sampling, stop sequences, tool choice behavior—these are operational knobs that affect output semantics, not just performance. They're often set in environment variables or config files that don't go through the same change management as application code, creating a category of behavioral changes that are nearly invisible to standard monitoring.

The operational implication: deployment rollback is no longer a single operation. When something goes wrong, you need to independently ask: did the prompt change, did the model change, or did configuration change?

Failure Modes SRE Runbooks Were Never Designed For

Classic microservice failures are visible. A 500 error generates a span in your trace. A timeout increments a counter. A queue backs up, memory climbs, disk fills. These failures produce signals that map cleanly to alert thresholds and runbook steps.

LLM failures are often semantically silent. The service returns 200. Token usage looks normal. Latency is within bounds. The model just produced something subtly, confidently wrong.

Semantic failures produce no error signals. Research on LLM-generated outputs has found that more than half of incorrect results are syntactically valid—the code compiles, the JSON parses, the response follows the schema. The system executed correctly; it solved the wrong problem. This creates a failure class that never fires a conventional alert. Detection requires either downstream failures (which lag the root cause by hours) or purpose-built evaluation monitoring that checks output quality on a sample of live traffic.

Non-determinism invalidates standard canary metrics. In a canary deploy for a traditional service, you compare error rates, p99 latency, and a few key business metrics between baseline and canary. If they're statistically equivalent, you promote. This assumes that identical inputs produce equivalent outputs. LLM outputs vary by definition. Two requests with the same input may differ in structure, tone, or correctness. Standard two-sample statistical tests on traditional metrics won't catch a regression in output quality unless you instrument semantic evaluation as a first-class metric alongside error rate.

Cost is an incident class, not just a budget concern. When an agent enters an unintended loop or a poorly bounded retrieval step pulls excessive context, token consumption can grow exponentially without triggering any service-level alert. Analysis of production LLM deployments identifies agent runaway cost as an active incident category happening to teams without budget guardrails. Unlike CPU saturation (which degrades performance and becomes self-limiting), unconstrained token usage can generate five-figure charges in minutes with no visible service impact until the invoice arrives or a provider rate limit kicks in.

Rate limit cascades behave differently than service overload. Data from early 2026 showed that 60% of LLM call errors in production were caused by rate limit exhaustion, not service failures. Rate limiting in LLM APIs is per-token, not per-request, which means burst traffic from a single verbose prompt can consume more quota than hundreds of short requests. Standard circuit breakers that count HTTP 429s will respond too slowly; effective rate limit management requires proactive token budget tracking across all concurrent requests, not reactive response-code monitoring.

What Transfers From Classical SRE

Not everything needs to be rebuilt. Several SRE fundamentals apply directly, with adaptation at the metric level.

SLO methodology transfers. Error budgets, burn rates, and the practice of writing SLOs before features ship all apply. What changes is the SLO surface: you're tracking not just availability and latency but accuracy (measured by sampling and eval), cost per request, and hallucination rate. Adaptive SLO thresholds—where the acceptable quality floor adjusts based on task type or user tier—are a natural extension of the burn-rate model.

On-call structure and postmortem culture transfer. Incident ownership, escalation paths, blameless retrospectives, and the five-whys process all remain valuable. The postmortem template needs new fields: which component changed (prompt, model, configuration), what made the failure invisible to standard monitoring, and what eval or behavioral signal would have caught it earlier.

Canary deployment logic transfers. The principle of routing a small fraction of traffic to a new version before full rollout applies directly. The implementation differs: canary traffic for LLM services should also run shadow evaluation (comparing outputs against the baseline asynchronously), and consistent user routing within sessions is essential because mixing model versions mid-conversation creates incoherent context.

Runbook structure transfers, but the steps change. The format of a runbook—trigger conditions, diagnostic steps, mitigation actions, escalation path—translates. The content is different for every failure category listed above.

The Three SRE Disciplines That Need to Be Rebuilt

Testing. Regression testing for deterministic systems assumes identical inputs produce identical outputs; a failure is unambiguous. LLM regression testing requires eval suites that measure semantic similarity against reference outputs, validate output format correctness, check safety policy adherence, and flag meaningful behavioral drift across prompt or model versions. These eval suites need to be fast enough to run in CI and comprehensive enough to catch regressions that don't surface as format errors. Most teams don't have them at all when they first ship an LLM feature, which means the first several incidents are also the first time they understand what to test for.

Incident diagnosis. When a microservice misbehaves, you pull traces, correlate error spikes with deploy timestamps, and bisect to a code change. When an LLM feature misbehaves semantically, none of those signals exist. The diagnostic workflow is different: identify whether output format changed (parse failure rate), whether semantic drift is detectable (embedding distance from golden outputs), whether the failure correlates with a prompt deploy, a model update, or an input distribution shift. This requires instrumentation that most teams don't set up until after their first invisible incident.

Observability instrumentation. Traditional observability serves one audience: the on-call engineer looking for a root cause. LLM observability serves at least four: the on-call engineer (service health), the ML engineer (model behavior), the product manager (quality and user impact), and finance (cost attribution). These audiences ask different questions from the same telemetry, which means a single dashboard doesn't serve any of them well. Teams that treat LLM observability as an extension of their existing APM setup typically find it inadequate when the first real quality regression occurs.

New Runbook Categories You Don't Have Yet

Teams operating microservices maintain runbooks for the incidents they know can happen. LLM services require runbooks for scenarios that have no equivalent in classical operations:

Prompt rollback. When a prompt change causes a regression, rolling back means reverting to the prior prompt version, not redeploying application code. This requires that prompts be version-controlled and externalized from the binary, that there's a known-good state to roll back to, and that someone on call has authority and tooling to execute the revert without a full deploy cycle. Teams that embed prompts directly in code and have no prompt registry typically discover this gap mid-incident.

Model version pinning and upgrade validation. If you're calling a provider-managed endpoint, you may not control when the underlying model changes. Runbooks for model updates should include: how to detect a model version change (version headers, behavioral probing), how to evaluate whether the new version is regression-free against your specific use cases, and how to pin to a prior version or activate a fallback if the new version is incompatible. This is analogous to dependency pinning, but the "dependency" is a cloud service with its own upgrade schedule.

Semantic regression triage. When monitoring surfaces a quality drop—output format success rate declining, semantic distance from reference outputs increasing, downstream parsing failures rising—the triage steps for a semantic regression differ from those for a service error. The runbook needs to distinguish: is this a prompt change regression, a model behavior change, an input distribution shift, or a retrieval quality degradation? Each has a different mitigation path, and confusing them wastes hours.

Cost runaway response. When token consumption exceeds expected bounds, the immediate response is different from CPU saturation. Token budget limits need to be enforced at the request level, not just monitored at the aggregate level. The runbook should specify: at what threshold to alert, how to throttle or shed load without breaking user sessions, how to identify which feature or endpoint is the source, and when to switch from a frontier model to a smaller fallback to reduce per-token cost while the incident is investigated.

Building the AI Ops Discipline

The teams running LLM services most reliably in production are not the ones with the best SRE practices—they're the ones who recognized early that their SRE practices needed extension rather than just application. They built eval-based CI gates before they had incidents that made them obvious. They externalized prompts into versioned registries before they needed to roll one back at 2am. They set up cost budget limits before an agent loop ran unchecked.

The gap between AI ops and classical platform engineering isn't a matter of degree; it's a matter of kind. The deployment unit is different, the failure modes are different, the diagnosis path is different, and the rollback semantics are different. None of that makes SRE principles wrong—it makes them necessary but not sufficient. The organizations building reliable LLM services treat AI ops as a distinct practice with its own runbooks, its own monitoring surfaces, and its own testing discipline. The ones treating it as an extension of their existing microservices platform will run the experiment, and their postmortem documents will document exactly which assumptions didn't hold.