[Paper Review] Public code for Model Collapse

Ilia Shumailov, Zakhar Shumaylov|arXiv (Cornell University)|May 27, 2023

Topic ModelingComputer Science152 citations

TL;DR

The paper shows that training on model-generated data causes a degenerative ‘model collapse’ where the true data distribution is forgotten, tails vanish, and learned models converge toward simplistic representations, across GMMs, VAEs, and LLMs.

ABSTRACT

Stable Diffusion revolutionised image creation from descriptive text. GPT-2, GPT-3(.5) and GPT-4 demonstrated astonishing performance across a variety of language tasks. ChatGPT introduced such language models to the general public. It is now clear that large language models (LLMs) are here to stay, and will bring about drastic change in the whole ecosystem of online text and images. In this paper we consider what the future might hold. What will happen to GPT-{n} once LLMs contribute much of the language found online? We find that use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear. We refer to this effect as Model Collapse and show that it can occur in Variational Autoencoders, Gaussian Mixture Models and LLMs. We build theoretical intuition behind the phenomenon and portray its ubiquity amongst all learned generative models. We demonstrate that it has to be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of content generated by LLMs in data crawled from the Internet.

Motivation & Objective

• Motivate the study of how generative model data can pollute subsequent models when data are repeatedly scraped from the web.
• Characterize the model collapse phenomenon across different model families (GMMs, VAEs, LLMs).
• Quantify how finite sampling and functional approximation errors drive divergence from the original data distribution.
• Argue the importance of access to genuine human-generated content to mitigate collapse.

Proposed method

• Introduce the concept of model collapse as a degenerative, generational learning process where each model trains on data produced by the previous one.
• Use Wasserstein distance (W2) to quantify divergence between the original distribution and generation-i distributions.
• Present simple toy models (discrete distributions and a single-dimensional Gaussian) to analytically bound risk and illustrate how tails erode over generations.
• Derive that both statistical approximation error and functional approximation error contribute to cascade effects, with superlinear data collection needed to keep distance finite.
• Extend analysis to GMMs and VAEs showing tails wash out and modes entangle over generations.
• Discuss fine-tuning in LLMs and note early signs of collapse when training from scratch versus fine-tuning.

Experimental results

Research questions

• RQ1 Does training on data generated by previous model generations cause the original data distribution to erode over generations?
• RQ2 Is model collapse observable across multiple generative model families (GMMs, VAEs, LLMs)?
• RQ3 What are the primary mechanisms (statistical vs functional approximation errors) driving collapse, and how can they be quantified?
• RQ4 What role does access to genuine human-generated data play in mitigating model collapse, especially for LLMs?

Key findings

• Model collapse is a universal degenerative process where generated data pollute subsequent training sets and cause models to misperceive reality.
• Tails of the original distribution disappear early, and over generations the learned distribution becomes more delta-like with reduced variance.
• In simple Gaussian and discrete-distribution toy models, finite sampling induces a random-walk-like drift that increases the distance from the true distribution unless sampling grows superlinearly.
• Experiments on GMMs and VAEs show that with generational data, the latent representations become unimodal and progressively diverge from the original modes.
• LLMs trained or fine-tuned on generated data exhibit early signs of collapse, highlighting risk in large-scale web data reuse without human-generated content.
• Theoretical bounds based on Wasserstein distance indicate that the expected deviation from the true distribution grows with generations under finite sampling, reinforcing inevitability of collapse without immense data regeneration.