The Near-Duplicate Filter That Took Your Only Hard Example With It

Your dedup step reported a corpus shrink of 28% and the training run finished six hours faster. The eval numbers came in flat-to-slightly-better. Nobody opened the diff of what got removed. Three weeks later support starts paging about a class of refund-reversal tickets the model used to handle and now flatly mishandles. There are eleven training rows that touched that exact pattern. Nine of them are gone — collapsed into a single representative that kept the shortest, cleanest phrasing and dropped the messy hostile-tone variants where the model had actually learned to de-escalate. Your dedup pipeline did that, and your evals did not catch it, because by the time the eval set was built, those examples were already gone from the train set the eval was sampled from.

This is the failure mode that bothers me about deduplication as a pipeline step: it presents itself as hygiene and it is actually distribution editing. Removing exact duplicates of boilerplate is hygiene. Removing near-duplicates by a similarity threshold is a sampling decision dressed up as one. The threshold picks which slices of your training distribution survive, and the slices most likely to lose are the ones where you have the fewest examples to begin with — which are also, almost by definition, the ones you were keeping for coverage rather than count.

The literature on this is louder than the discourse implies. The original "Deduplicating Training Data Makes Language Models Better" result from 2022 measured exact and approximate duplicates and found real gains, and that result got generalized into a folk rule that more dedup is more better. SemDeDup pushed it further — drop 50% of LAION via semantic similarity, retain performance, sometimes improve OOD — and that became a default. Then FairDeDup looked at what happened to subgroup metrics when you ran SemDeDup on a vision-language corpus and found that the same pipeline that retained aggregate accuracy degraded gender-bias and skin-tone metrics. Same algorithm, same data, different scoring lens, opposite verdict. That gap is where production teams keep getting hurt.

Dedup is selection, not cleaning

The mental model worth replacing is that dedup removes redundant copies and the model is unaffected because it has the information already. That story holds for exact duplicates of boilerplate text. It breaks the moment your similarity function decides that two examples that look 0.82 alike are "the same."

Near-duplicate filtering imposes an equivalence relation on your dataset, then keeps one representative per equivalence class. Two design decisions are buried in that sentence. The first is what counts as the same — your shingling, your MinHash bands, your embedding model, your similarity threshold. The second is which representative survives — shortest, longest, most central in the cluster, earliest by timestamp, highest by a quality score. Neither decision is neutral, and neither gets reviewed the way an architecture change or a loss function change would.

The asymmetry that bites you is that frequent patterns dominate their own clusters and survive in a thousand variants, while rare patterns cluster tightly around one or two examples and survive in zero variants. After dedup, the dataset looks balanced on aggregate token count and is more imbalanced than before on coverage of the long tail. This is the regression that does not show up in eval until production traffic includes the slice that lost its representatives.

Where the rare slices actually live

A dedup pipeline does not target rare slices on purpose. It targets them as a structural consequence of how rare slices are shaped in real datasets. A few patterns recur:

• Tightly clustered phrasings. Disability-access complaints, accessibility-tooling bug reports, regulatory-disclosure clarifications. These topics have a finite vocabulary, get reported in similar ways, and look like duplicates to a similarity function. The dominant interpretation survives; the discriminative variants are dropped.
• Hostile or non-standard register. A polite "I would like a refund please" and an angry "give me my money back" are behaviorally distinct from a model-training standpoint — the de-escalation behavior is learned from the second, not the first. A dedup pipeline keyed on lexical overlap of the noun phrases happily collapses them.
• Dialect, code-switch, and transliteration. Mixed-script inputs and dialect variants get dedup-collapsed against the standardized form because preprocessing normalized them on the way in. The model loses the noisy-input robustness it needed for exactly the users whose inputs are noisy.
• Time-separated recurrence. The same outage report appears in March, June, and October. A dedup pipeline that drops by content hash and ignores timestamps loses the recurrence signal — that this is a chronic issue, not a one-off — which is precisely the signal a model needs to escalate.
• Hard negatives near positives. In ranking and classification, the most useful examples are the near-misses: looks like the positive class, is actually the negative class. A similarity-based dedup happily merges those two clusters and you lose the contrast the model was learning the boundary from.
• Labeler disagreements. Two annotators look at the same prompt and label it differently because it is genuinely ambiguous. Dedup keeps one. The model now learns the dataset has more certainty than it actually does, and gets miscalibrated on the cases that were ambiguous to humans.

None of these are exotic. They are the long tail of every real dataset that has been collected from real users.

The eval set lies because it came from the same distribution

The usual answer to "is the dedup safe" is "the evals are green." This is the part of the workflow that quietly fails. If you sample your eval set from the same corpus before dedup, run dedup on the train set, and grade on the eval set, you are testing on slices that may also be over-represented in the train set. If you sample your eval set after dedup, you are testing on the same distribution you trained on, with the same slices missing.

Either way, the eval reports on what the dedup left you, not on what production traffic will send. The slice you lost is exactly the slice no longer represented in either set. A green eval on an evaluation set that has been edited by the same hand that edited the training set is not a green eval, it is a tautology.

The fix is not subtle but it is annoying enough that few teams do it. You need an eval set that was sampled before dedup, stratified by the slices you care about — explicitly enumerated, not implicit — and you need to grade on those slices separately. A dedup pipeline that drops 28% of the corpus and keeps the same per-slice recall on the eval is fine. A dedup pipeline that drops 28% of the corpus, holds aggregate accuracy, and tanks one slice from 84% to 41% is the failure mode, and you cannot see it without slice-broken-out scoring.

What the dedup pipeline should be telling you

Dedup output is usually a single number — corpus size before, corpus size after, percent reduction. The information you actually need is the histogram of what got removed.

A dedup pipeline worth running in production should be emitting:

• Cluster size distribution. How many clusters of size 1 (no near-duplicates, kept), size 2–10, size 100+. Big clusters are where the boilerplate lives. Small clusters are where the long tail lives. A pipeline that flattens small clusters at the same rate as big ones is over-correcting on the tail.
• Per-slice retention rate. Define your slices ahead of time — by topic, by source, by language, by user-cohort, by sentiment, by whatever your domain actually cares about — and report the retention rate per slice. The slice that drops from 1,200 examples to 18 needs a human review before you ship that corpus.
• Representative-choice audit. When a cluster collapses to one example, log which one was kept and why. A spot-check of "the kept representative is the shortest one" across a hostile-tone cluster is enough to find the bug where you trained politeness into a model whose users are not all polite.
• Diff against the previous corpus. Run the same dedup on last quarter's data with this quarter's settings and report the slice-level delta. A threshold change from 0.85 to 0.80 sounds small. It can move per-slice retention by 30 points.

If your dedup step does not produce these reports, it is making sampling decisions you cannot see, which means you cannot review them, which means the next regression will look like a model regression and not a data regression and the wrong team will spend a week chasing it.

Treat dedup as a modeling decision, not a hygiene chore

The instinct to dedup harder comes from the right place — wasted training compute on the same example a hundred times is real waste, and the original 2022 result was real. The mistake is treating "more dedup is better" as a free lunch that scales with threshold sensitivity. It does not. There is a regime where dedup is hygiene — exact and very-near matches of templated text. There is a regime where it is selection — semantic-embedding clusters of behaviorally distinct examples. The transition between those regimes is exactly where teams set their thresholds because the compute savings look great there. That is also exactly where the long tail starts disappearing.

A few patterns that hold up in practice. Run separate dedup policies for the head and the tail of your distribution — boilerplate is safe to crush, rare-slice examples are not. Keep frequency counts when you collapse, so the model can still learn that this pattern is common even if you only feed it one copy. Pick the representative deliberately rather than letting "shortest" or "first" win by default — for a hostile-tone cluster you want the hostile-tone example, not the one that survived a politeness rewrite. Make the threshold a tuned hyperparameter with an eval that scores on rare-slice recall, not a default lifted from a paper that was trained on a different distribution for a different objective.

And the one that takes the most discipline: when you fix a production regression that traces back to a missing slice, do not just patch the eval set. Patch the dedup policy that removed it, and add a regression test that catches that policy from removing the same shape of example again. The eval patch buys you one slice. The policy patch buys you the next ten you have not had paged on yet.

The dedup step that erases your only canonical edge case is not a bug in MinHash or in SemDeDup. It is a category error about what dedup is for. The day your dedup pipeline starts emitting per-slice retention reports — the day a human signs off on which slices were allowed to collapse — is the day it stops being a silent capability regression and starts being a reviewable decision. The pipeline has not gotten any smarter. You have just stopped letting it grade itself.