The Model API Is Tier 0 Now. Design the Degraded Mode Before the Status Page Turns Red

Ask an infrastructure team what happens if the primary database goes down and you will get a rehearsed answer: replicas, failover runbooks, RTO and RPO numbers someone signed off on. Ask the same team what happens if the model API goes down and you will usually get a shrug and a link to the provider's status page. That asymmetry made sense in 2023, when the LLM powered an experimental sidebar. It stopped making sense the day your support flow, your search ranking, your code review bot, and your onboarding assistant all started routing through one vendor's inference endpoint.

The model API is now a tier-0 dependency for a lot of products — revenue-critical, sitting in the request path next to the database — but most disaster-recovery plans still treat it like a nice-to-have integration. The result is a familiar incident shape: the provider degrades, every AI feature in the product throws the same spinner, on-call stares at a status page they can't influence, and nobody can answer the only question that matters: what is this product supposed to do right now?

That question has to be answered in an architecture review, not during the incident. This post is about what a real degraded mode looks like for an AI dependency, why the obvious answer — "just fail over to another provider" — is much harder than it sounds, and what to decide before the status page turns red.

You Inherited a Dependency Class You Don't Control

Databases fail in ways your team can debug. Model APIs fail in ways you can only observe. In November 2025, a configuration bug in Cloudflare's bot management system took down a huge slice of the internet for about three hours — and with it ChatGPT, Perplexity, and Claude, essentially every major hosted AI assistant at once. None of those providers had done anything wrong. Their shared edge dependency failed, and every product downstream of them failed too, transitively, invisibly, and simultaneously.

That incident exposed the uncomfortable shape of the dependency graph: your product depends on a model provider, which depends on a CDN, a cloud region, and a handful of inference clusters spread across different hardware platforms. Anthropic's own postmortem of its August–September 2025 incidents described three overlapping infrastructure bugs that intermittently degraded response quality — not availability — across the different hardware backends serving Claude. That failure mode is worth staring at: the API returned 200s the whole time. Your health checks passed. Your users got worse answers.

So a tier-0 AI dependency actually has three distinct failure modes, and your degraded-mode design has to cover all of them:

• Hard down: 5xx errors, timeouts, the status-page-is-red scenario. Easy to detect, painful but honest.
• Brownout: elevated latency and error rates, intermittent 429s and 529s. Detection is a thresholding problem, and naive retries make it worse.
• Silent degradation: the API is up, latency is normal, and the outputs are subtly worse. No traditional monitor catches this; only output-quality evals sampled in production do.

Traditional DR planning assumes failures are binary and detectable. The model API gives you neither guarantee.

Multi-Provider Failover Is Not Multi-Region Failover

The reflexive answer to all of this is a gateway with a fallback chain: primary model, then a cheaper sibling from the same provider, then a competitor, then maybe a self-hosted open-weights model as the last resort. Gateways and routers have made the mechanics genuinely easy — health checks, circuit breakers that trip after a handful of failures and probe again after a cooldown, automatic rerouting in milliseconds instead of waiting out timeouts.

The mechanics are the easy part. The hard part is that a database failover and a model failover are semantically different operations. When you fail a Postgres primary over to a replica, the replica speaks the same SQL, holds the same data, and returns the same rows. When you fail GPT over to Claude, or Claude over to Llama, you get a system that accepts the same request format and returns something different.

Anyone who has actually swapped models in production knows swapping is not plug-and-play. Prompts co-adapt to the model they were tuned against: every edge-case fix an engineer made to the system prompt encoded that specific model's interpretation quirks. The same prompt that yields clean structured JSON from one model produces markdown-wrapped JSON from another and unsolicited commentary from a third. Tool-calling conventions, refusal boundaries, and instruction-following under long context all differ.

A fallback path you have never evaluated is not a fallback — it is a different product that activates precisely when nobody is watching quality.

Mid-conversation failover is harder still, and this is where the multi-region analogy fully breaks down. A database replica shares state with the primary through replication. Model providers share nothing. If a user is twelve turns into a session with an agent when the primary goes down, failing over means replaying the accumulated context — conversation history, tool results, cached system prompts — into a model that interprets all of it slightly differently.

Prompt caches don't transfer. Fine-tunes don't transfer. The new model may re-litigate decisions the old model already made, contradict its earlier answers, or take a different tool path halfway through a workflow. There is no "replication lag" metric for the model now has a different personality.

This doesn't mean multi-provider failover is worthless. It means it must be treated as a pre-evaluated degraded mode with its own quality bar, not a transparent redundancy layer. Concretely: maintain a per-task eval suite, run it against the fallback chain on a schedule, and know before the incident which tasks survive failover acceptably and which should be disabled instead of degraded.

Degraded Mode Is a Product Decision Wearing an Infrastructure Costume

The most useful reframe: degraded mode is not one system-wide switch, it is a per-feature decision about what the product does when inference is unavailable or untrusted. For each AI-powered surface, there are roughly four options, in descending order of ambition:

• Serve from cache. A meaningful fraction of real traffic concentrates on recurring intents, which is why semantic caching — matching new queries to previous answers by embedding similarity rather than exact text — routinely absorbs a large share of requests in production deployments. A cache that saves you money on Tuesday is also the thing that keeps FAQ-shaped queries answered during Thursday's outage. But a cache used as a fallback needs stricter rules than a cache used as an optimization: confidence thresholds so borderline matches fall through to an error state instead of a wrong answer, and TTLs so you don't serve stale policy or pricing information with a straight face.
• Fall back to the pre-LLM implementation. The keyword search that the RAG pipeline replaced, the rules-based triage that the classifier replaced, the template that the drafting assistant replaced. Teams delete these predecessors at their peril. The old system was worse than the model on the average case, but it is infinitely better than a spinner. If the predecessor is already gone, a small self-hosted model serving a simplified version of the task plays the same role: an answer of last resort with no external dependency.
• Queue the work. Not every AI task is interactive. Summarization, enrichment, categorization, report generation — these can absorb hours of provider downtime invisibly if the architecture treats them as queued jobs with deadlines rather than synchronous calls. The degraded mode for async work is simply a longer queue, which is the cheapest degraded mode ever designed. During an outage, this is also where load-shedding happens: batch and background traffic stops competing with interactive traffic for whatever capacity remains.
• Say so, honestly. For the genuinely irreplaceable features, the right degraded mode is an explicit state: "The assistant is unavailable; here's the manual path." An honest unavailable-state with a link to the non-AI workflow preserves more trust than a chatbot that hangs for ninety seconds and then hallucinates through a smaller model it was never evaluated on. Users forgive outages. They do not forgive being lied to about them.

The exercise that makes this concrete: inventory every AI call site in the product and assign it one of these four modes, plus a tier. What you will usually find is that only a small minority of call sites are genuinely tier-0 interactive — and that the honest label for several "critical" AI features is queue it or turn it off. That inventory shrinks the real failover problem to a size where the pre-evaluated fallback chain from the previous section becomes tractable: you only need cross-provider quality parity on the handful of surfaces that truly cannot wait.

The Business-Continuity Question to Answer Before the Incident

Classic BCP asks two numbers of every dependency: how long can we be down (RTO), and how much data can we lose (RPO). For a model dependency, data loss mostly isn't the issue — the provider holds no state you can't reconstruct. The questions that actually matter are different, and they are business questions that engineering cannot answer alone:

Which revenue path breaks, and after how long? An AI coding product with a down model API has no product. A CRM with a down AI summarizer has an annoyance. Most products sit in between, and pretending everything is the first case produces failover over-engineering, while pretending everything is the second produces the all-features-spin incident. Tier-0 designations are supposed to be reserved for revenue- or safety-critical paths; AI features have been grandfathered into products without anyone re-running that classification.

What quality floor are we willing to ship under a red banner? Failover to a weaker model is a decision to serve worse answers under your brand. For a support deflection bot, a small accuracy drop during a three-hour outage is invisible. For anything touching money, health, or legal exposure, the degraded mode is probably off, and someone in the business needs to have agreed to that in advance — because during the incident, the pressure will all point toward "keep it running on whatever model responds."

Who pays for the standby capacity? Real degraded modes cost real money before any incident: a second provider contract kept warm with a trickle of traffic so its quality is continuously measured, provisioned capacity or a reserved deployment in a second region, eval suites run against the fallback chain on every prompt change. Provider SLAs, where they exist at all, refund a percentage of spend — they do not refund your lost revenue. The gap between those two numbers is the budget argument for standby capacity, and it is winnable, but only if someone makes it before the outage rather than in the retro.

Then rehearse it. Mature continuity programs run failover exercises on tier-0 systems once or twice a year; almost nobody does this for the model dependency. A game day where you black-hole the primary provider's endpoint in staging will teach you more in two hours than this article can — usually starting with the discovery that the circuit breaker works but the feature flags to disable non-critical AI surfaces don't exist, so the fallback provider immediately drowns under 100% of traffic that should have been shed.

The Status Page Is Not a Strategy

There is a version of this story where nothing described here gets built, and it is honestly defensible for some products: if your AI features are genuinely decorative, the degraded mode of "wait it out" is fine, and the November 2025 outage was a three-hour inconvenience. But that defense has to be made explicitly, because the default trajectory points the other way. Every quarter, more of the product's core loops route through inference. The dependency quietly graduates from nice-to-have to tier-0, and nobody re-runs the DR review, because no single launch felt like the moment it changed.

So run the review now, while it's cheap. Inventory the call sites. Assign each one a degraded mode — cache, predecessor, queue, or honest absence. Pre-evaluate the fallback chain on the few surfaces that earn it. Put quality evals, not just uptime checks, on the primary, because the ugliest failure mode returns 200s. And get a business owner to sign the quality floor and the standby budget before the incident, not during it.

The model API will go down again. That part is not under your control. Whether your product has an answer to "what do we do now" — that part always was.