[Paper Review] A Tale of Tails: Model Collapse as a Change of Scaling Laws

Elvis Dohmatob, Yunzhen Feng+3 more|arXiv (Cornell University)|Feb 10, 2024

Topic Modeling4 citations

TL;DR

This paper introduces a theoretical framework linking model collapse in large language models to changes in scaling laws when training data becomes increasingly synthetic. By analyzing how AI-generated data truncates or narrows the tails of natural data distributions, the authors demonstrate that scaling laws degrade through loss of power-law behavior, leading to performance decay, skill unlearning, and eventual model collapse—validated through large-scale experiments on Llama2 and a transformer on an arithmetic task.

ABSTRACT

As AI model size grows, neural scaling laws have become a crucial tool to predict the improvements of large models when increasing capacity and the size of original (human or natural) training data. Yet, the widespread use of popular models means that the ecosystem of online data and text will co-evolve to progressively contain increased amounts of synthesized data. In this paper we ask: How will the scaling laws change in the inevitable regime where synthetic data makes its way into the training corpus? Will future models, still improve, or be doomed to degenerate up to total (model) collapse? We develop a theoretical framework of model collapse through the lens of scaling laws. We discover a wide range of decay phenomena, analyzing loss of scaling, shifted scaling with number of generations, the ''un-learning" of skills, and grokking when mixing human and synthesized data. Our theory is validated by large-scale experiments with a transformer on an arithmetic task and text generation using the large language model Llama2.

Motivation & Objective

• To understand how scaling laws in large language models change when training data contains increasing amounts of synthetic data.
• To investigate the mechanisms—such as tail truncation and narrowing—through which model collapse emerges from synthetic data in the training distribution.
• To analyze the degradation of model performance, including loss of emergent skills and failure to maintain power-law scaling.
• To validate theoretical predictions with large-scale experiments on Llama2 and a transformer model trained on an arithmetic task.
• To explore the conditions under which grokking or recovery of performance may occur when mixing synthetic and real data.

Proposed method

• Theoretical analysis of scaling laws under heavy-tailed input distributions (e.g., Zipf’s law) and their breakdown under truncated or narrowed distributions from AI generation.
• Modeling AI generation via top-p (nucleus) sampling and temperature scaling, which systematically truncate or narrow the tails of the next-token distribution.
• Empirical evaluation using Llama2-7B fine-tuned on Wikitext-103 chunks, with successive generations of synthetic data from prior models to simulate iterative synthetic data accumulation.
• Tracking performance decay via cross-entropy loss and perplexity on test sets, comparing models across generations and data mixing ratios.
• Introducing controlled data mixing (e.g., 90% synthetic, 10% real) to study recovery phenomena and grokking effects.
• Analyzing combinatorial effects on sequence-level metrics like perplexity, showing how tail narrowing leads to spurious performance degradation.

Experimental results

Research questions

• RQ1 How does the inclusion of synthetic data, generated via LLMs, alter the scaling laws of subsequent models?
• RQ2 What mechanisms—tail truncation or tail narrowing—drive the degradation of scaling behavior in models trained on synthetic data?
• RQ3 Can model collapse be predicted and explained through changes in the statistical properties of training data distributions?
• RQ4 Under what conditions does performance recovery (e.g., grokking) occur when mixing synthetic and real data?
• RQ5 To what extent do emergent skills, such as arithmetic reasoning, degrade or disappear as model training shifts toward synthetic data?

Key findings

• Synthetic data generation via top-p sampling and temperature scaling systematically truncates or narrows the tails of natural data distributions, disrupting the power-law scaling behavior essential for model improvement.
• As generations of synthetic data accumulate, scaling laws degrade: test loss no longer follows a power-law decay with data size, leading to performance stagnation or decline.
• Models trained on multiple generations of synthetic data exhibit loss of emergent skills, such as arithmetic reasoning, due to the loss of rare but critical data patterns.
• Experiments with Llama2-7B show measurable performance decay over generations, with loss increasing and perplexity rising as data becomes more synthetic.
• Mixing synthetic data with real data at a 90:10 ratio leads to a grokking-like recovery of performance, suggesting that a small amount of real data can restore scaling behavior.
• Theoretical analysis confirms that tail narrowing in next-token distributions leads to combinatorial degradation in sequence-level metrics like perplexity, even without direct data distribution shift.