2 posts tagged with "fallback"

The fallback path you wrote nine months ago — the one that catches model timeouts, swaps to a cheaper provider, returns a templated message when both are down — has not actually run in production for the last twelve weeks. It was exercised once during the original launch, the integration tests still pass against it, and the runbook still references it. None of that means it works. A refactor in week six changed the shape of the upstream context object. A library bump in week nine quietly moved a config key. The code still compiles. The tests still pass because they were written against the same stale fixtures the code was. The next time your primary path 504s, your "graceful degradation" will throw a NullPointerException into a user's face, and the postmortem will note — for the third time this year — that the fallback was never re-tested after the upstream contract changed.

This is the quiet failure mode of resilience engineering in AI systems. The fallback path is the part of your application that exists specifically to be ignored. Production traffic flows around it for ninety-nine days out of a hundred. CI never exercises it because no test was ever wired to. The team that owns it forgets it exists between incidents. Then on day one hundred, when the primary model provider has a regional outage and you finally need it, the path bit-rots in front of a paying customer.

The first time I watched a fallback router cause a worse incident than the outage it was trying to mitigate, the postmortem document had a header that read: "Primary degraded for 11 minutes. Fallback degraded our parser for 6 days." Nobody had written code wrong. Nobody had skipped the schema review. The integration tests against the secondary provider had been green when the fallback was wired up, eighteen months earlier. What had happened in between was that one of the two providers had quietly tightened its enum coercion policy, and the contract our downstream parsers had been written against — a contract we believed was "JSON Schema, more or less" — had drifted from a shared standard into two slightly incompatible dialects.

This is the failure mode I keep seeing, and it keeps surprising teams that should know better. "JSON mode" sounds like a feature you turn on. It is not. It is a contract you maintain — separately, against every provider you might route to — and the contract drifts every quarter as vendors evolve their structured-output stacks. The "drop-in replacement" your provider docs gestured at when you signed the contract is, in production, a maintained translation layer whose absence converts your fallback path into a paper compliance artifact: present in the architecture diagram, broken on the day you needed it.