The Six-Month Cliff: Why Production AI Systems Degrade Without a Single Code Change

Your AI feature shipped green. Latency is fine, error rates are negligible, and the HTTP responses return 200. Six months later, a user complains that the chatbot confidently recommended a product you discontinued three months ago. An engineer digs in and discovers the system has been wrong about a third of what users ask — not because of a bad deploy, not because of a dependency upgrade, but because time passed. You shipped a snapshot into a river.

This isn't a hypothetical. Industry data shows that 91% of production LLMs experience measurable behavioral drift within 90 days of deployment. A customer support chatbot that initially handled 70% of inquiries without escalation can quietly drop to under 50% by month three — while infrastructure dashboards stay green the entire time. The six-month cliff is real, it's silent, and most teams don't have the instrumentation to see it coming.

Three Forces That Erode Production AI Silently

Silent degradation in LLM-powered systems is rarely one thing. It's typically the convergence of three independent decay processes happening simultaneously, each contributing incrementally until the aggregate drop becomes impossible to ignore.

Eval distribution drift is the most underappreciated. When you launch an AI feature, you build an evaluation set reflecting the queries and edge cases you can imagine. But production traffic evolves. Users adopt new terminology, ask about features you've added since launch, and frame questions in ways your eval set never anticipated. The model hasn't changed, but the gap between what you're testing and what's actually happening in production widens every week. Your eval suite stays green while real-world accuracy quietly erodes — and you have no signal because you're measuring the wrong thing.

Silent base-model updates are the more treacherous cause because they're completely outside your control. Model providers regularly update the behavior of their models without changing the model name you're calling in production. Anthropic dropped version pinning support, forcing users onto whatever the latest version is. OpenAI offers dated snapshots but regularly deprecates them on 3-6 month cycles. When a model provider ships a behavioral update — even a well-intentioned one — your prompts can break in subtle ways: structured output formats change slightly, refusal patterns shift, reasoning style changes enough to affect downstream logic. One commonly reported incident: a silent model update doubled CI failure rates over three days as agent behavior shifted without a single line of code changing.

Knowledge base rot compounds both. Any RAG system is only as fresh as its indexed documents. Pricing pages drift from reality within weeks. Compliance documents expire and aren't re-indexed. The system that indexed your documentation at launch continues confidently answering questions based on information that's months out of date — and since RAG outputs still look coherent and well-formatted, there's no obvious signal that the underlying facts have become wrong. One analysis found that RAG systems lose roughly a third of their effective accuracy within 90 days, purely due to knowledge staleness.

Why Your Monitoring Won't Catch It

The deepest problem with the six-month cliff is that traditional observability was designed for infrastructure failures, not semantic ones. HTTP status codes, latency percentiles, and error rates all look healthy while your AI feature tells users the wrong things. The gap between "the system is up" and "the system is useful" is where silent degradation lives.

Only 62% of organizations running AI agents in production can inspect what their agents actually do at each individual step, despite 89% claiming to have observability in place. Observability means request/response logging. It doesn't mean quality measurement.

LLM-specific failures compound this problem in ways that traditional ML monitoring doesn't anticipate. In classic machine learning, you can detect drift using statistical methods on input feature distributions — KL divergence, population stability indices, feature importance shifts. These methods are partially useful for LLM systems, but they're blind to semantic drift: two responses can be statistically similar in token length and vocabulary while encoding completely different (and one of them wrong) meanings. A model that used to accurately summarize complex technical content may start producing plausible-sounding summaries that omit critical caveats — and no token-distribution metric will catch that.

Multi-agent systems face an additional compounding failure. Over extended operation, agents encounter input distributions increasingly divergent from what they were designed for. Decision-making patterns progressively deviate from design specifications without explicit parameter changes or any observable failure event. The most dangerous production response is a silent 200 OK: the request completed, the response looks reasonable, and the data quality has been silently poisoned for 11 days before anyone noticed.

What Production-Grade AI Maintenance Actually Looks Like

The correct mental model is to treat AI feature maintenance like infrastructure maintenance: scheduled, measured, and mandatory — not something you do in response to user complaints.

Golden set testing is the foundation. A golden set is a fixed collection of inputs with expected outputs (or evaluation rubrics) that you run continuously against production traffic samples. A minimum viable golden set has 50-100 examples covering your core use cases. A production-ready set runs 200-500 examples. A mature system continuously adds examples drawn from production failures. The test runs on every deployment candidate; a 3% drop in overall quality score relative to the main branch triggers an investigation, not a ship.

Four statistical signals catch drift before users do:

• KL divergence on response length distributions: Track rolling 7-day baselines. Response length changes are a leading indicator for quality shifts in roughly 87% of cases — models that start being more or less verbose are usually changing their behavior more broadly.
• Embedding drift: Track the semantic similarity of responses to expected outputs over time. When embeddings start drifting from your reference distribution, something has changed in how the model interprets your prompts.
• LLM-as-judge scoring: Use a secondary LLM call to score production outputs against a rubric for correctness and groundedness. Run this on a 1% sample of production traffic continuously.
• Refusal fingerprinting: Monitor changes in refusal patterns. A sudden spike in refusals, or a drop, is almost always a signal that the model's behavioral policy has shifted — often due to a silent provider update.

Refresh cadence should be explicit, not reactive. The standard schedule that production-grade teams converge on:

• Prompt re-evaluation: Every 30 days, or within 48 hours of any model provider announcement, run your full eval suite against production traffic samples from the past week. Prompts that passed at launch often need adjustment as models update.
• Knowledge base audit: Every 60-90 days, audit document freshness. For RAG systems, track "document age" as a first-class metric alongside retrieval latency and answer quality. Any document older than its expected shelf life (days for pricing, months for policy, a year for stable architecture content) needs verification before it's surfaced to users.
• Golden set expansion: Every quarter, add 50-100 examples from production failures and edge cases discovered in the past period. A golden set that doesn't grow becomes an increasingly partial view of what production actually looks like.
• Model version audit: Track every model version you call in production and the provider's deprecation timeline. For OpenAI, this means watching the deprecations page and building 60-day migration runways into planning. For Anthropic, it means accepting that updates are automatic and investing in rapid eval infrastructure instead.

The Organizational Failure Behind the Technical Failure

The six-month cliff is partly a technical problem and partly an organizational one. When an AI feature ships, the team that built it moves on. The feature is "done." Ownership dissolves. Nobody owns the eval suite, nobody owns the knowledge base refresh, and nobody has monitoring in place because the feature was shipped — not because it was maintained.

The teams that avoid the six-month cliff have made one structural decision: they treat AI features as requiring ongoing maintenance on par with production services. That means:

• Assigning explicit ownership for eval suite maintenance and knowledge base refresh, separate from the team that built the feature
• Treating "AI quality" as an SLO alongside latency and availability — with alerts, thresholds, and on-call responsibility
• Scheduling quarterly reviews of model provider announcements and adjusting version pinning and migration timelines accordingly
• Treating silent failures — outputs that look correct but encode stale or wrong information — as first-class bugs, not "expected behavior"

The engineering discipline here isn't glamorous. It's the difference between a CI pipeline that runs before every deploy and one that was set up once and never extended. Golden sets, drift monitors, and refresh cadences are the equivalent of integration tests for AI systems: the first thing teams skip when they're moving fast and the thing they desperately wish they'd invested in when production quietly falls apart.

Where to Start

If you have an AI feature in production today and no ongoing maintenance infrastructure, the priority order is:

1. Build a golden set this week. Even 50 examples covering your top use cases gives you a regression detector. Run it before your next deploy.
2. Add LLM-as-judge scoring to 1% of production traffic. You don't need a perfect rubric. A simple "is this response correct and grounded?" prompt run on a sample gives you a quality trend line.
3. Audit your knowledge base age. Find the documents in your RAG index that haven't been updated since launch. Assume they're wrong about at least some things.
4. Check your model version exposure. For each model you call in production, find out when the provider last updated its behavior and what the deprecation timeline is.

The six-month cliff is not inevitable. It's the default outcome when AI features are treated as one-time deliverables rather than production systems that require continuous measurement and maintenance. The teams that discover degradation first — before users do — are the ones who built the instrumentation to see it.