Your Users Drift While Your Model Stands Still

Six weeks after launch, your quality dashboard starts sagging. Thumbs-down rates creep up, task completion drifts down, and the on-call channel fills with screenshots of bad responses. The team does what teams do: they diff the prompts (unchanged), check the model version (pinned), audit the retrieval index (fresh), and bisect the deploy history (nothing shipped). Everyone concludes the model provider silently degraded the model. The provider, of course, insists nothing changed.

Everyone is looking in the wrong place. Nothing in the system changed. The users did.

Launch-week metrics assume launch-week users. But people adapt to an AI product within weeks, and they adapt in ways that systematically break the assumptions baked into your prompts, your evals, and your launch benchmarks. Your model is frozen. Your users are not. The gap between them is a form of drift that most teams don't instrument for at all — and it produces the most confusing incident pattern in AI engineering: metric decay with no deploy.

The Three Ways Users Drift

User adaptation isn't random noise. It follows predictable patterns, and each one degrades a different part of your system.

Power users compress. Early on, users write the careful, context-rich requests your demo data was full of: "I'm planning a trip to Tokyo in March with two kids, we like food and museums, can you suggest a 5-day itinerary?" Three weeks in, the same user types "tokyo 5d kids redo march." They've learned the product usually gets it, so they stop paying the typing tax.

Their private jargon — abbreviations, elliptical follow-ups, references to conversations the model can't see — becomes the dominant input distribution. None of it was in your eval set, because none of it existed when you built your eval set. Your prompts were tuned on complete sentences; production is now fragments.

Skeptics probe. Some fraction of your users treats your guardrails as a puzzle. This is not hypothetical: when ChatGPT launched, jailbreak techniques were circulating on Reddit and Discord within days, and researchers have since shown that systematic prompt collections can bypass safety filters in nearly every major model.

You don't need adversarial researchers for this effect, either. Ordinary users learn which phrasings trip your refusal logic and route around them — sometimes to do something harmless your classifier is too jumpy about, sometimes not. Either way, the traffic hitting your guardrail model in month three is meaningfully more adversarial than the traffic it was calibrated on in month zero.

Everyone else relaxes. The novelty effect is one of the best-documented phenomena in product analytics: engagement spikes when a feature is new, then decays as the newness fades. Microsoft observed interactions with a Copilot feature drop roughly 64% over nine weeks; longitudinal studies of AI workflows show the same U-shaped curve, where usage falls off after the first month unless the tool becomes genuinely load-bearing.

What survives the novelty cliff is different traffic, not just less traffic. The curious explorers churn out; the users with real, repetitive jobs remain. Your input mix shifts from "show me something cool" to "do my Tuesday task again," and the model's performance on the second population is what your metrics now measure.

Each of these is invisible if all you track is aggregate quality scores. The score moves, and nothing you own explains why.

Why This Gets Misdiagnosed as Model Drift

The standard drift taxonomy — data drift, concept drift, model drift — was built for systems where the model is the thing that changes. In an LLM application with a pinned model version, teams reach for these categories and grab the wrong one.

The confusion is understandable because the symptom is identical: outputs get worse against a fixed rubric. But the mechanism matters enormously for the fix. If the model degraded, you escalate to the provider or swap models. If the retrieval index went stale, you fix the pipeline. If the users drifted, both of those are wasted motion — the model is answering exactly as well as it ever did, on a question distribution it was never evaluated against.

There's a second, nastier layer: user drift and system drift interact. A user who got one bad answer starts writing longer, more defensive prompts with explicit instructions ("do NOT summarize, list every item"). Those defensive prompts are themselves out-of-distribution and can trigger new failure modes, which produce more defensive prompting. You can watch a cohort's prompt style curdle over a month in response to a single bad week. If you only look at output quality, this reads as gradual model decay. If you look at inputs, it's clearly a feedback loop between user trust and prompt style.

The diagnostic question that cuts through it: has the input distribution moved? That question has a concrete, measurable answer, and most teams never ask it because all their monitoring points at outputs.

Instrument the Inputs, Not Just the Outputs

Output monitoring — eval scores, feedback rates, refusal counts — tells you that something changed. Input monitoring tells you who changed, and it's the only way to separate user drift from model drift.

The good news is that the machinery already exists; it's just usually pointed at documents instead of users. Embed a sample of production prompts from a stable baseline window, embed the current window, and compare the distributions — Jensen-Shannon divergence or Wasserstein distance on clustered embeddings both work, and language-model embeddings capture the semantic shifts that word-frequency methods miss. When the distance metric jumps, you have evidence the traffic changed before you burn a week auditing prompts that didn't.

Beyond raw semantic drift, a few cheap proxies specifically catch adaptation, as opposed to just new topics:

• Prompt length and completeness per returning-user cohort. Compression shows up as a steady decline in median prompt length among your oldest cohorts, while new-user prompts stay long. If you only track the global median, cohort mixing hides the signal.
• Session-relative references. Count prompts that are unintelligible without prior context ("same as before but for Q3," "redo it the other way"). Rising anaphora means users have built a mental model of your product's memory — one your context assembly may not actually honor.
• Guardrail trigger composition. Track not just refusal rate but the diversity of phrasings arriving at your safety layer. Narrowing diversity with a rising bypass rate means users are learning the boundary.
• Cohort-split quality metrics. Every quality metric you own should be splittable by user tenure. A score that's flat for week-one users and falling for week-eight users is user drift by definition — the model is doing the same thing; the veterans are asking harder.

None of this requires new vendors. It requires deciding that the input distribution is a first-class production signal, with a baseline, a dashboard, and an alert — the same treatment you give latency.

Your Eval Set Has a Shelf Life

The deeper implication is uncomfortable: an eval set is a snapshot of how users talked to your product at one moment in its life. Its expiry date is set by user habituation, not by the calendar.

Teams intuitively understand that eval sets go stale when they ship new features or enter new markets. What they miss is that the eval set goes stale even if the product doesn't change, because the users keep learning. The careful, well-formed queries you curated at launch stop being representative the moment your power users learn to compress. An eval suite that's 95% green can coexist with a degrading production experience simply because it's testing a dialect nobody speaks anymore.

The fix is a refresh loop, and the industry has converged on its shape: continuously sample production traces, auto-curate the interesting ones (high disagreement, negative feedback, novel clusters in embedding space), annotate them, and fold them into the eval set while retiring cases that no longer match any live traffic cluster. Teams that derive test cases from production report substantially better real-world performance than teams evaluating on synthetic or launch-era data alone — production-derived cases are the only ones guaranteed to be in-distribution.

Two disciplines make the loop trustworthy. First, version the eval set like code, so a score change is attributable to either a system change or an eval change, never silently both. Second, when you refresh, keep the old set around and run both for a cycle — the gap between the frozen launch-era eval and the refreshed one is itself a clean measurement of how far your users have drifted.

A useful heuristic for refresh cadence: your eval set should turn over at roughly the rate your user base does. High-growth product with weekly cohorts of new users? Monthly refresh is the floor. Stable enterprise deployment with the same 500 seats? Quarterly may be fine — but watch the compression metrics, because a stable user base habituates faster, not slower.

Design for the Second Month, Not the First Week

Once you accept that users will drift, some of it stops being a monitoring problem and becomes a design opportunity.

Compression, for instance, is not an attack — it's a compliment. Users compress because they trust the product to fill in the gaps. You can meet them halfway: persist their stable preferences so "tokyo 5d kids" actually resolves against known context, and have the model ask one targeted clarifying question when a fragment is genuinely ambiguous instead of guessing. The teams that treat terse prompts as malformed input end up fighting their best users; the teams that treat them as a feature request for memory end up with a stickier product.

Similarly, the post-novelty traffic shift is information about product-market fit. The queries that survive the 64% engagement cliff are your product's real job description. Re-tuning prompts and re-weighting evals toward the surviving workload — rather than the launch-week fantasy workload — is some of the highest-leverage prompt engineering available, and it requires no new capabilities at all.

The general principle: launch is the start of the distribution shift, not the end of development. Plan for a second calibration pass at week six as a scheduled milestone, the same way you'd plan a post-launch retro. Budget for it. The teams that get surprised by metric decay are the ones whose roadmap implicitly assumed the users of month three would behave like the users of day one.

Models get all the attention in drift discussions because they're the exotic new component. But the model is the most stable thing in your stack — it's pinned, versioned, and reproducible. The humans on the other side of the text box are none of those things. Instrument them accordingly.