Model Collapse Starts in Your Own Data Lake

Everyone worries about model collapse as an internet-scale problem: AI slop floods the web, the next generation of foundation models trains on it, and quality decays in a slow civilizational feedback loop. That framing is comforting because it makes collapse someone else's problem — a thing that happens to OpenAI and Anthropic, on a timescale of years, mitigated by armies of data-cleaning PhDs.

Here is the uncomfortable version: the same feedback loop is already running inside your company, and it converges much faster than the internet-scale one. Every logged completion that gets labeled as a "gold example," every model-written document that lands in your RAG corpus, every LLM-judged eval that promotes an LLM-generated answer — each is a small act of training on your own exhaust. You don't need nine generations of recursive pretraining to feel it. In a production system that mines its own logs for few-shot examples and fine-tuning data, generation two ships next quarter.

The original research on this is stark. Shumailov et al.'s 2024 Nature paper showed that models trained indiscriminately on recursively generated data suffer irreversible defects: the tails of the original distribution vanish first, then outputs converge toward repetitive, low-variance mush. Their famous demonstration started with text about medieval architecture and ended, nine generations later, with a list of jackrabbits. But the lab setup — full replacement of training data each generation — is not what makes the finding relevant to you. What makes it relevant is the mechanism, and the mechanism doesn't care whether the loop runs through the public internet or through your data lake.

The Flywheel You Built on Purpose

Nobody sets out to train on their own outputs. What teams set out to build is a data flywheel: log every interaction, harvest the good ones, and feed them back to make the product better. Vendor architecture diagrams present this as an unambiguous virtue — usage generates data, data improves the model, a better model drives more usage.

The problem is that "the good ones" are increasingly model-authored. Trace where the recycled data actually comes from:

• Few-shot examples mined from logs. A prompt engineer searches production traces for high-rated completions and pastes the best into the system prompt. The model is now imitating itself, with the flattering ones selected in.
• Fine-tuning on accepted outputs. The classic flywheel: user clicked "accept," so the completion becomes a training pair. But "accepted" mostly means "not annoying enough to reject" — a much weaker signal than "correct," and one biased toward the model's existing style.
• RAG corpora that eat agent output. Your agents write tickets, summaries, runbooks, and wiki pages. Those documents get indexed. Next week the model retrieves its own summary as authoritative context and summarizes it again, one lossy compression stacked on another.
• LLM judges grading LLM candidates. When the same model family generates the answer and scores it, you've built preference amplification, not evaluation. Recent work on LLM-powered feedback loops shows systems drifting from learning human preferences to reinforcing model-induced ones.

Each hop looks defensible in isolation. The compounding is what kills you: a completion becomes a doc, the doc becomes retrieval context, the retrieved answer gets a good rating, the rating promotes it into the fine-tune set. Four hops, zero of them labeled "synthetic."

Collapse Arrives as Confidence, Not Gibberish

The jackrabbit example makes collapse sound easy to spot. It won't be. A 2025 study of recursive synthetic training identified what it calls knowledge collapse: factual accuracy deteriorates while surface fluency persists, or even improves. The model doesn't degrade into word salad — it gets more polished and more confidently wrong at the same time, because fluency is exactly the property that self-training reinforces and factual grounding is exactly the property it erodes.

That failure mode is nearly invisible to standard monitoring. Your perplexity looks fine. Your style adherence improves — of course it does, the model is being graded against its own prose. Format-compliance evals go green. Meanwhile three quieter things degrade:

• Tail knowledge goes first. Statistical sampling error means rare events — the unusual customer configurations, the edge-case error codes, the questions asked once a month — fall out of the recycled distribution before anything else. These are precisely the cases where you needed the model most.
• Diversity narrows. Outputs converge toward the model's modal phrasing and modal reasoning path. Users notice the sameness before your metrics do.
• Errors calcify into facts. A hallucinated parameter name, written into a runbook, retrieved as context, restated in five more documents, is no longer a hallucination. It's corpus consensus. Your provenance-free data lake cannot tell you which came first.

The macro numbers say the ambient contamination is rising too: Ahrefs' analysis of 900,000 newly created web pages found roughly three-quarters contained at least some AI-generated content, and tracking studies show AI-assisted articles reaching parity with purely human-written ones since late 2024. Whatever you scrape, license, or paste from the web is already partially synthetic. Your internal loop stacks on top of that baseline.

The Research Nuance That Actually Helps

If the story ended with "recursion destroys everything," the only advice would be despair. It doesn't. The follow-up literature draws a distinction that should directly shape your pipeline design: what matters most is whether synthetic data replaces real data or accumulates alongside it.

Gerstgrasser et al. showed that when each generation trains on the accumulated pool — original human data plus all synthetic generations — test error stays bounded no matter how many iterations you run. Collapse in the strong sense requires the replacement regime, where fresh model output crowds out the original distribution. Later statistical work ("Collapse or Thrive?") refines this further: outcomes depend on the ratio of real to synthetic data and on whether anyone verifies the synthetic portion before it re-enters training. And a 2025 line of work on synthetic-data verification shows that even imperfect verifiers — filters that catch only some of the junk — substantially delay or prevent degradation.

Translated out of theorem-speak, the pipeline implications are concrete:

1. Never let recycled outputs displace your human-authored anchor set. Keep the original corpus immutable and always in the mix. Deletion-and-replacement is the collapse regime; accumulation is the safe one.
2. Ratio is a dial you must actually monitor. If you can't state what fraction of your current fine-tune set is model-generated, the honest answer is "more than last quarter, and rising."
3. Verification changes the regime. Human review, execution checks, or even a weaker independent filter between generation and re-ingestion buys you disproportionate protection. The unfiltered flywheel is the pathological case from the papers.

None of these mitigations is exotic. All of them require one capability most data lakes lack entirely: knowing which bytes came from a model.

Provenance Is the Whole Game

Here's the uncomfortable audit question: for a random document in your RAG index or a random row in your training warehouse, can you say whether a model wrote it? For most teams the answer is no — not because the information was hard to keep, but because nobody kept it. The moment an LLM output is pasted into a ticket, merged into a wiki, or written to a warehouse table, its origin evaporates.

There is no low-background steel for text; you cannot reconstruct pre-contamination purity after the fact. Provenance is only cheap at write time.

The discipline that works looks like lineage tracking applied to generation, and it has three layers:

• Tag at the source, not at ingestion. Every model-generated artifact gets provenance metadata the instant it's created: which model, which prompt version, what human review it received. Your inference gateway is the natural chokepoint — it already sees every completion, so make it stamp every completion. Downstream tagging is archaeology; source tagging is bookkeeping.
• Propagate the tag through transformation. A summary of a synthetic document is synthetic. A human edit of a model draft is mixed — and "mixed" is worth tracking as its own class, since studies of new web content find heavily human-edited AI text is the dominant category, not pure generation. If your lineage tooling (the same kind you use for GDPR data-subject requests) can answer "which tables descend from this source," it can answer "which rows descend from a model."
• Quarantine before re-entry. Synthetic and mixed content lands in a staging zone by default, excluded from training-set queries and RAG indexing until it passes an explicit gate — human verification, execution against ground truth, or at minimum an independent filter that wasn't produced by the same model family. The key property is that re-entering the training supply is a decision someone makes, not a side effect of being stored.

Then verify the loop is actually broken the way collapse actually manifests: hold out an evaluation set of tail cases — rare intents, low-frequency entities, the questions your support team sees monthly, not hourly — frozen from before your flywheel spun up, and watch it specifically. Aggregate benchmarks stay green deep into knowledge collapse; the tail eval is where the first real signal shows up.

Treat Model Output Like an Untagged Isotope

The teams that get this right share a posture, not a tool: they treat model-generated text as a material that's perfectly useful but must never enter the food supply unlabeled. Synthetic data isn't poison — the accumulation results show that clearly, and deliberate, verified synthetic data is behind some of the best post-training results of the past two years. Unlabeled synthetic data is the poison, because it removes your ability to choose the regime you're in. With provenance, you're running controlled data augmentation. Without it, you're running the Shumailov experiment on your own product, in the replacement regime, with quarterly generations.

The practical starting point is one afternoon of work: make your inference gateway stamp every completion with provenance metadata, and add a synthetic flag to your ingestion schema with a default of "unknown, quarantined." Everything else — propagation rules, ratio dashboards, tail evals — builds on those two bits.

The teams that skip them will spend 2027 trying to un-ring the bell, digging through their data lake for pre-model strata the way materials scientists once salvaged pre-atomic shipwrecks for uncontaminated steel. The ones that start tagging now will never have to.