What 99.9% Uptime Means When Your Model Is Occasionally Wrong

A telecom company ships an AI support chatbot with 99.99% availability and sub-200ms response times — every traditional SLA metric is green. It is also wrong on 35% of billing inquiries. No contract clause covers that. No alert fires. The customer just churns.

This is the watermelon effect for AI: systems that look healthy on the outside while quietly rotting inside. Traditional reliability SLAs — uptime, error rate, latency — were built for deterministic systems. They measure whether your service answered, not whether the answer was any good. Shipping an AI feature under a traditional SLA is like guaranteeing that every email your support team sends will be delivered, without any commitment that the replies make sense.

The gap matters because AI features are increasingly load-bearing for B2B products. When an AI system generates an invoice summary, classifies a contract clause, or drafts a compliance checklist, "it returned HTTP 200" is not a useful success criterion. The customer cares whether the output was correct, and they will hold you accountable for it — even if your contract doesn't.

Why Traditional SLAs Break for AI

Traditional SLAs work for deterministic systems because success is binary and observable. A database either returns a row or it doesn't. An API either responds within 300ms or it doesn't. You can measure it, log it, and alert on it. The SLA reflects ground truth.

AI output quality is none of those things. The same prompt can produce responses that range from excellent to factually wrong, depending on phrasing, context window state, model version, temperature, and dozens of other variables. Success is a distribution, not an event. And critically, detecting failure often requires a human in the loop — or a second AI evaluation pass — because the system returns a confident-sounding 200 with a hallucinated answer.

There are also second-order problems. Uptime SLAs measure availability, not quality degradation. A model provider can roll out a new version that reduces your task-specific accuracy by 15% while keeping p99 latency unchanged. Every dashboard is green; your product silently gets worse. Aggregate benchmarks compound this — a model that scores well on general benchmarks may perform significantly worse on your specific task distribution, and you won't discover this from vendor SLA metrics alone.

The legal exposure compounds the engineering problem. Courts have begun treating AI-generated outputs as factual representations in the context they were delivered. If your AI tool generates a financial summary that a customer relies on, hallucinations are not just a quality problem — they can be a breach of contract or misrepresentation problem. And 88% of AI vendors cap their own liability at monthly subscription fees, meaning the downstream risk lands on you.

What an Honest AI Quality Commitment Looks Like

The answer is not to promise perfect outputs. It is to define what "good enough" means for a specific task, and to make that commitment explicit and measurable.

The basic structure of an honest AI quality commitment has three parts:

A task scope definition. The quality floor applies to a named task with defined inputs, not to the model in general. "The system will correctly extract invoice date, vendor name, and total amount with ≥ 95% accuracy on standard invoice formats" is a defensible commitment. "The AI will be accurate" is not.

A measurement protocol. How do you know when the floor has been breached? Sampling-based evaluation with human or LLM judges, applied on a defined cadence (weekly spot-checks on 1% of production traffic), gives you a repeatable measurement method. Ad hoc internal assessments do not.

A remediation trigger. When quality drops below the floor, what happens? Escalation to human review, rollback to a prior model version, customer notification — these need to be defined in advance, not negotiated after a customer complaint.

For internal SLOs, the frame is similar but the emphasis shifts to enabling teams to detect degradation before customers do.

Designing Internal SLOs That Actually Catch Quality Failures

Most engineering teams that adopt SLOs for AI features make the same mistake: they measure latency and error rate because those are easy to instrument, and they assume quality is someone else's problem.

A useful quality SLO has three properties. First, it measures something that correlates with task success, not just request completion. For an extraction task, this might be field-level accuracy on a sample of outputs evaluated against ground truth. For a summarization task, it might be a relevance score from an LLM judge, with periodic human calibration to verify the judge is tracking reality. For a classification task, it is precision and recall on a held-out test set that is refreshed quarterly as the input distribution shifts.

Second, it uses an error budget rather than a hard threshold. An error budget treats quality headroom as a finite resource. If your quality floor is 92% accuracy on invoice extraction, and your last 30-day sample is at 94%, you have 2 percentage points of budget to spend on experimentation, model rollouts, and edge cases before you breach the commitment. When the budget is low, you stop experimenting and stabilize. When it is high, you have room to take risk. This is the same framing SRE teams use for uptime — it aligns incentives without requiring zero tolerance for failure.

Third, it catches version drift. Model providers update their models, sometimes with no warning. Prompt behavior shifts. Input distribution changes as your user base grows. A quality SLO that only runs at launch will not catch the slow degradation that occurs over months. Shadow evaluation — routing a small slice of production traffic to an evaluation pipeline that scores outputs against your quality criteria — gives continuous visibility into whether your quality floor is still being met, independent of vendor-side metrics.

The Measurement Window Problem

One subtlety that bites teams in practice: the measurement window for quality SLOs is fundamentally different from the measurement window for latency SLOs.

Latency SLOs can be evaluated on every request. A quality SLO based on human evaluation can realistically only be evaluated on a sample, with a lag. If your evaluation cadence is weekly, you might not detect a quality regression until it has been running for 5 days. That is not a failure of the SLO design — it is an honest reflection of the cost of measurement. The design implication is that you need leading indicators that fire faster: automated LLM-judge scoring on 100% of outputs, with the human evaluation as a calibration layer that confirms the judge is still correlated with your actual quality standard.

The other measurement window problem is burst versus sustained failure. A traditional window-based SLO treats a 2-minute degradation the same as a 2-hour one, as long as both occupy a single window. For AI quality, burst failures matter more than they look — a 10-minute window where 60% of outputs are wrong can do more customer damage than a 48-hour window where 8% are wrong. Your SLO design should distinguish between sustained degradation below the floor and acute accuracy events, and your remediation triggers should be different for each.

The B2B Contract Language That Actually Works

When AI features are part of a B2B contract, the quality commitment needs to be explicit and bounded. Vague language — "the system will provide accurate responses" — creates open-ended liability that neither side can evaluate.

The patterns that hold up under scrutiny:

Defined task scope with in/out-of-scope carve-outs. The commitment covers identified task types (invoice extraction, not general document Q&A). Out-of-distribution inputs are explicitly excluded from the quality guarantee. This prevents customers from testing edge cases that were never part of the scope and then claiming SLA breach.

Measurement-based quality floors. "≥ 92% field-level accuracy on samples evaluated against customer-provided ground truth, measured monthly" is auditable. "High accuracy" is not. The contract should specify who conducts the evaluation, how ground truth is established, and what constitutes a valid sample.

Escalation triggers and remediation timelines. If the measured accuracy falls below the floor, what is the SLA for remediation? A reasonable structure: notify the customer within 48 hours of detection, provide a root-cause analysis within 5 business days, and implement a fix or rollback within 14 days. Without explicit timelines, "we're working on it" is the default.

Explicit model-change notification. If you are using a third-party model, changes to that model can affect your quality commitments. Your contracts should require you to notify customers before major model version changes and re-validate quality on their task distribution before cutover.

What these clauses share is that they make the quality commitment specific, measurable, and time-bounded — the same properties that make uptime SLAs enforceable. The goal is not to promise perfection; it is to give customers something concrete they can evaluate and give your team something concrete they can build toward.

Building the Internal Infrastructure

None of this works without the infrastructure to measure it. The minimal stack for a team shipping a quality-committed AI feature:

An evaluation dataset. A set of input/output pairs with human-verified ground truth, representative of your production task distribution, refreshed at least quarterly. This is the hardest part to build and the first thing teams skip. Without it, you cannot compute accuracy — you can only compute token counts.

An automated scoring pipeline. A system that evaluates a sample of production outputs against your quality criteria, with results written to a dashboard. For tasks where ground truth is available, this is direct comparison. For tasks where it is not (summarization, drafting), this is an LLM judge with human calibration on a subset.

A drift alert. A threshold on the automated scoring that fires when your observed quality drops below a warning level — say, 94% when your floor is 92%. The warning level gives you time to investigate before you breach the contractual commitment.

A model change runbook. A documented process for evaluating quality on your task distribution before any model version change goes to production. This should be treated with the same rigor as a database schema migration.

The Mindset Shift

The deeper change is not technical — it is about how engineering teams think about what they are promising when they ship an AI feature.

For deterministic systems, shipping means the system works. For probabilistic systems, shipping means the system works well enough, often enough, for a defined set of inputs. The "well enough" and "often enough" need to be quantified before you commit to a customer, not after they complain.

This is uncomfortable because it forces you to admit, explicitly, that your system will sometimes be wrong. But the alternative — shipping under a vague quality promise and handling failures ad hoc — is worse. Customers who sign explicit quality floor agreements know what they bought. Customers who feel misled by implicit accuracy promises are the ones who churn and litigate.

The teams that are shipping durable AI products in 2026 are the ones that treated quality commitments as a first-class engineering problem from the start: designed the measurement infrastructure before the launch date, negotiated contract language before the customer asked, and built the error budget into their release process before the first model version change. That discipline is not a constraint on shipping AI features — it is what makes AI features shippable at all.