[Paper Review] Scale Dependent Data Duplication

Joshua Kazdan, Noam Levi+7 more|arXiv (Cornell University)|Feb 18, 2026

Data Quality and Management0 citations

TL;DR

The paper demonstrates that semantic duplicates become increasingly harmful at scale, derives scaling laws to account for limited semantic uniqueness, and offers practical methods to estimate effective semantic pool size to restore predictive scaling.

ABSTRACT

Data duplication during pretraining can degrade generalization and lead to memorization, motivating aggressive deduplication pipelines. However, at web scale, it is unclear what constitutes a ``duplicate'': beyond surface-form matches, semantically equivalent documents (e.g. translations) may induce redundant training signals once models become sufficiently capable. Practically, this means that semantic duplicates operate increasingly like exact duplicates during training. We present evidence that duplication is scale-dependent in two ways. First, as model capability increases, cross-entropy loss gradients for semantically equivalent documents become more aligned. Smaller models, by contrast, produce gradients that reflect surface similarity (e.g., shared tokens) rather than semantic similarity. Second, we embedded all 192 million FineWeb-Edu-Dedup documents using EmbeddingGemma-300m. For moderate corpus sizes, the cosine similarity between nearest-neighbors follows an isotropic power law baseline. However, as corpus size grows to hundreds of billions of tokens, the nearest-neighbor similarities deviate sharply, indicating accelerated semantic collisions. Finally, controlled pretraining on data sampled with replacement from pools of finite unique documents shows that limited uniqueness yields mild degradation for small models, but rapidly increasing loss penalties for larger models, breaking naive scaling extrapolation. We derive explicit scaling laws that allow practitioners to estimate deviation from expected scaling due to limited semantic uniqueness of the pretraining corpus. Our results identify and resolve an unstudied source of scale-dependence, allowing for more accurate prediction at scale.

Motivation & Objective

• Quantify how semantic duplicates affect training signals as model capability grows.
• Show that near-duplicate signals emerge for semantically equivalent documents at scale.
• Demonstrate that larger corpora exhibit accelerated semantic collisions beyond surface-similarity expectations.
• Derive scaling laws that incorporate limited semantic uniqueness to restore predictable scaling.
• Provide a practical method to estimate effective semantic pool size from training data statistics.

Proposed method

• Measure gradient similarity between per-document cross-entropy gradients and semantic-preserving transformations across model scales.
• Embed a large real-world document set (FineWeb-Edu-Dedup) and analyze nearest-neighbor cosine similarities across corpus sizes to identify scaling breaks.
• Train decoding transformers with controlled data pools to observe how finite uniqueness degrades performance as compute increases.
• Develop a theory where semantics are hierarchical latents and define effective duplicates via gradient decomposition (mu, delta_z, xi_x).
• Propose a three-parameter plane law Delta(C,K) = a C^beta K^(-gamma) to restore predictable scaling when uniqueness is limited.
• Provide a method to estimate an effective K_eff from mean nearest-neighbor cosine similarity (Equations 29–34).

Figure 1 : Semantic-preserving transformations yield more aligned gradients for larger/stronger models. We sample $N{=}1000$ FineWeb-Edu-Dedup documents and compute per-document gradients of normalized next-token cross-entropy (Eq. 2 ) for each model. We report mean cosine similarity between (i) unr

Experimental results

Research questions

• RQ1 Do semantically equivalent documents induce more aligned training gradients as model capability increases?
• RQ2 How does corpus size influence semantic collisions and deviation from isotropic scaling laws?
• RQ3 Can we model and correct for scale-dependent degradation due to limited semantic uniqueness?
• RQ4 How can we estimate an effective semantic pool size from observable training streams to restore scaling predictability?
• RQ5 Do synthetic data corpora show the same collapse of scaling laws as real data, and what does that imply for data diversity?

Key findings

• Semantic duplicates induce aligned gradient updates in more capable models, making semantic duplicates act like exact duplicates during training.
• Nearest-neighbor cosine similarity scaling deviates from power-law baselines at large corpus sizes, indicating accelerated semantic collisions.
• Synthetic data show scaling law deviations earlier than real data, suggesting less semantic diversity in synthetic datasets.
• Finite uniqueness in training data causes scale-dependent degradation, breaking naive scaling extrapolations for larger models.
• A three-parameter plane law accurately predicts eval losses across compute and pool sizes, restoring scalable predictivity.
• An effective semantic pool size K_eff can be estimated from mean nearest-neighbor cosine similarity, enabling practical scaling corrections.

Figure 2 : Semantic sensitivity emerges over training and is accelerated by scale. For a fixed model family, we compute AUC to detect whether a candidate gradient corresponds to a semantic-preserving transformation of the same document versus an unrelated document, with cosine similarity to the orig