The Pager Doesn't Ring for a 3% Regression: On-Call for Statistical Failures

Your on-call rotation was designed to catch a different kind of failure than the one that will actually take down your AI product. It watches for the service that stops responding, the latency curve that spikes, the error rate that crosses 1%. These are step functions: something was working, then it wasn't, and the discontinuity is loud enough to page a human at 3 a.m. The entire apparatus — thresholds, runbooks, escalation policies — assumes failure announces itself.

The failure mode that matters for a system with a model in the loop is the opposite. Last Tuesday your extraction pipeline was 94% accurate. This Tuesday it's 91%. Nothing crashed. Every request returned a 200. Latency is flat. The output is still well-formed JSON. But three percent of your users are now getting subtly wrong answers, and they will not file a bug, because the answer looks right. The pager stayed silent because there was nothing for it to fire on. By the time someone notices — usually a customer, usually angry, usually weeks later — the regression has been quietly compounding the whole time.

This is the gap nobody put in the on-call ladder. We hired SREs to keep deterministic systems up, and then we shipped probabilistic systems on top of them and assumed the same monitoring would transfer. It doesn't. A statistical failure is invisible to deterministic alerting by construction, and closing that gap requires rethinking what "an incident" even means.

Why Your Existing Alerts Are Blind to This

Deterministic alerting works because deterministic systems have a correct answer and an incorrect one. A 500 is wrong. A null pointer is wrong. A request that takes 30 seconds when the SLO is 200ms is wrong. You set a threshold at the boundary between acceptable and unacceptable, and the metric either crosses it or it doesn't. The signal is the crossing.

Model quality has no such boundary. There is no single response you can point to and say "this is the bug." The 91%-accurate model produces individually plausible outputs; the failure only exists in aggregate, as a shift in a distribution. You cannot write if accuracy < 0.92 then page because you usually don't have accuracy in real time — the ground-truth label that would tell you whether an answer was right arrives days later, if it arrives at all. When a human reviews the output next week, or the user churns next month, or never, the feedback that would let you compute accuracy is delayed or simply absent.

So you're left monitoring the two things that are available immediately — the inputs going in and the outputs coming out — and trying to infer quality from their shape. That's a fundamentally different discipline. You're not asserting correctness; you're detecting change. And change is a statistical claim, which means a single data point tells you nothing. One weird output is noise. A thousand outputs whose distribution has drifted is a signal. Your alerting has to operate on distributions, not events.

Worse, the cause is often outside your code entirely. Your provider silently updated the model weights behind the same endpoint. The version you pinned was deprecated and you got rolled forward. The mix of users shifted because marketing launched in a new region. None of these show up in a diff, a deploy log, or a stack trace. The system that changed isn't one you control, and the only evidence is a slow bend in a curve you weren't plotting.

The Two Distributions You Can Actually Watch

Since ground truth is delayed, you build your early-warning system out of the two signals that arrive in real time: the distribution of what goes in, and the distribution of what comes out.

Input drift asks whether the data hitting your model today looks like the data it was validated on. If your classifier was tuned on English support tickets and you're suddenly seeing 20% Spanish, the model hasn't degraded — but its effective accuracy has, because it's now operating off-distribution. The standard tool here is the Population Stability Index (PSI), which buckets a feature's distribution and compares today against a reference window. The conventional reading: below 0.1 is stable, 0.1 to 0.25 is worth watching, above 0.25 means the population has shifted enough to plausibly hurt the model. For continuous features, a Kolmogorov-Smirnov test answers the same question with a p-value; for categorical ones, chi-square does. None of these require a label. They only require that you saved a baseline.

Output drift asks whether your model's responses have changed shape, independent of whether the inputs did. This is the one that catches a silent provider update. If yesterday 8% of your outputs were refusals and today it's 15%, something moved — maybe the provider tightened a safety filter, maybe your prompt is interacting badly with a new model version. You don't need to know the cause to know the distribution shifted. For free-text outputs, the modern approach is to embed responses and watch the embedding distribution, because LLM drift usually lives in semantic space rather than in surface token counts. A clever low-cost variant: treat the model's first-token log-probabilities as samples and run a two-sample permutation test, which catches distributional shifts at a fraction of the cost of scoring every output.

The crucial mental shift is that both of these are leading indicators, not verdicts. Drift is a lead, not proof of failure — data can move while performance holds perfectly steady, and you'll page yourself into alert fatigue if you treat every PSI bump as an incident. The discipline is to use drift to decide where to look, then confirm with something closer to ground truth before you call it a regression.

Building the Confirmation Layer: Proxies and Sampled Evals

Drift tells you the weather changed. It doesn't tell you the crop died. To bridge from "something shifted" to "quality dropped," you need cheaper proxies for correctness that you can compute continuously.

The most practical is a sampled eval loop. You can't have a human grade every output, but you can sample a small fraction — say 1% — and score it, either with a rubric run by a separate judge model or, for the highest-stakes slices, by a human reviewer. This gives you a noisy but unbiased estimate of quality that updates daily instead of whenever labels happen to arrive. The sample is small enough to be affordable and large enough that a real 3% regression becomes statistically visible within a day or two rather than a quarter.

Layer onto that a set of guardrail proxies that correlate with quality and cost nothing to compute: schema-validation pass rates, refusal rates, output length distributions, the fraction of responses that fail a business-rule check, tool-call success rates for agents. None of these is quality, but each is a cheap canary. A sudden jump in malformed JSON or a creep in average response length is the kind of thing that precedes a quality drop and shows up instantly. Treat them as the smoke detectors that send you to the sampled eval for confirmation.

The architecture that emerges is a funnel. Cheap, instant, noisy signals (drift, guardrail proxies) at the top, firing often and routing attention. A more expensive, slower, more trustworthy signal (sampled evals) in the middle, confirming or dismissing. And true ground truth at the bottom, arriving late but anchoring the whole system so your proxies don't quietly decalibrate. Each layer exists to decide whether the next, costlier layer is worth invoking.

Slice Everything, Because the Aggregate Lies

Here is the trap that makes statistical regressions so dangerous: a 3% drop in overall accuracy can be a 30% collapse for one cohort, completely masked by everyone else holding steady. Your headline metric looks like a rounding error. For Spanish-speaking users on mobile, the product is now broken.

Aggregate monitoring is structurally blind to this. The only defense is to compute your metrics per slice — by locale, device, customer tier, input channel, model version, tenant — and alert on the slices, not just the global number. Slice-based monitoring is what localizes drift and surfaces the regression that lives inside a subgroup. The cost is that you now have dozens or hundreds of metric series, and naive thresholding across all of them will drown you in false positives from small, noisy segments.

This is exactly where statistical alerting earns its place over static thresholds. Instead of a fixed line, you flag a slice when its metric deviates significantly from its own historical behavior — a z-score against the slice's rolling baseline, a CUSUM chart that accumulates small persistent deviations until they're unmistakable. CUSUM is especially well-suited here: it's designed to catch exactly the small, sustained shift that a threshold misses, because it sums the drift over time rather than waiting for any single point to cross a boundary. A 3% regression that holds for a week is invisible to a threshold and screaming on a CUSUM chart.

Statistical alerting also adapts. Normal behavior for an AI product is not stationary — your traffic mix changes, your prompts evolve, your user base grows. A static threshold set in January is wrong by March. An adaptive baseline relearns "normal" continuously, so it alerts on genuine deviations rather than on the slow legitimate evolution of your system. The combination most teams land on is a hybrid: keep crude thresholds as a first line for the obvious, gross failures, and layer statistical, per-slice detection on top for the subtle ones.

What On-Call for Statistical Failures Actually Looks Like

Operationally, this changes the shape of the job. A deterministic incident is binary — it's firing or it's resolved — and the runbook is a sequence of steps to restore service. A statistical incident is a hypothesis: "quality may have dropped for segment X starting around time T." The on-call engineer's first job isn't to fix it, it's to confirm it's real, because drift flags are leads and a meaningful fraction will be false alarms.

So the runbook inverts. It starts with triage of the signal itself: is this a true distribution shift or sampling noise in a small slice? Then localization: which feature, which cohort, which time window? Then correlation with external events: did a provider version change, did we ship a prompt edit, did traffic composition move? Only then, mitigation: roll back the prompt, pin the previous model version, route the affected slice to a fallback, or kick off retraining. The skill being exercised is closer to data analysis than to firefighting, and the people you put on this rotation need to be comfortable reasoning about distributions, not just reading dashboards for red.

The cultural shift is the hardest part. Engineers trained on deterministic systems want a green checkmark that says "correct." Observability for statistical systems can't give you that — it can only show you change and help you reason about whether the change is bad. You are trading the comfort of a clear pass/fail for the discomfort of continuous probabilistic judgment. The teams that make this transition stop asking "is the model right?" and start asking "did the model get quietly worse for someone, and would I find out before they did?"

That question is the one your current pager can't answer. Build the layer that can — drift detection to route attention, sampled evals and guardrail proxies to confirm, per-slice statistical alerting to catch the regressions hiding inside the aggregate — and the 3% drop stops being the failure you discover from an angry customer three weeks late. It becomes the one you caught on Tuesday afternoon, while it was still 3%.