2 posts tagged with "disaster-recovery"

Ask an infrastructure team what happens if the primary database goes down and you will get a rehearsed answer: replicas, failover runbooks, RTO and RPO numbers someone signed off on. Ask the same team what happens if the model API goes down and you will usually get a shrug and a link to the provider's status page. That asymmetry made sense in 2023, when the LLM powered an experimental sidebar. It stopped making sense the day your support flow, your search ranking, your code review bot, and your onboarding assistant all started routing through one vendor's inference endpoint.

The model API is now a tier-0 dependency for a lot of products — revenue-critical, sitting in the request path next to the database — but most disaster-recovery plans still treat it like a nice-to-have integration. The result is a familiar incident shape: the provider degrades, every AI feature in the product throws the same spinner, on-call stares at a status page they can't influence, and nobody can answer the only question that matters: what is this product supposed to do right now?

The DR runbook your team rehearses every quarter was written for a stack you no longer fully run. It says: promote the replica, repoint DNS, drain the queue. It assumes state lives in databases, queues, and object storage — places the SRE org has owned, named, and tested for a decade. Then last quarter you shipped an agent. Working memory now lives in the inference provider's session cache, scratchpad files on a worker's local disk, in-flight tool results that haven't been written back, and a partial plan-and-act trace that exists only in the prompt history of one model call. None of that is on the asset register. None of it is in the runbook.

When the region drops, the agent doesn't fail cleanly. It half-completes. The user sees a workflow that started but the failover region cannot resume, the customer's invoice gets sent twice or not at all because the idempotency key lived on the dead worker, and the on-call engineer reads a Slack thread that begins "the orchestrator is up, but..." and ends six hours later with a credit-card chargeback queue.

This is the gap nobody named: agentic features have a state model the existing DR plan doesn't describe. The team that hasn't written that state surface down is one regional outage away from learning what their runbook's silence costs.