[Paper Review] Nepotistically Trained Generative-AI Models Collapse

Matyáš Boháček, Hany Farid|arXiv (Cornell University)|Nov 20, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper demonstrates that Stable Diffusion, a popular text-to-image diffusion model, suffers from severe model collapse when retrained on even 3% of its own generated images. The model rapidly degrades in image quality and diversity, with no full recovery even after subsequent retraining on real data, revealing a critical vulnerability to data poisoning via self-generated content.

ABSTRACT

Trained on massive amounts of human-generated content, AI-generated image synthesis is capable of reproducing semantically coherent images that match the visual appearance of its training data. We show that when retrained on even small amounts of their own creation, these generative-AI models produce highly distorted images. We also show that this distortion extends beyond the text prompts used in retraining, and that once affected, the models struggle to fully heal even after retraining on only real images.

Motivation & Objective

• To investigate the impact of retraining generative AI models on their own synthetic outputs, a phenomenon termed 'model poisoning'.
• To assess whether self-generated data leads to irreversible degradation in image quality and diversity in diffusion-based models like Stable Diffusion.
• To evaluate the effectiveness of retraining on real images to 'heal' models previously poisoned with synthetic data.
• To examine whether color matching or quality curation of self-generated data can mitigate the collapse effect.
• To explore the broader implications of data poisoning in the context of AI training data provenance and model robustness.

Proposed method

• Retrained the base Stable Diffusion model iteratively on mixed datasets containing varying ratios of real faces (from FFHQ) and self-generated faces.
• Used iterative retraining with a fixed number of steps per iteration, updating the model on each batch of mixed data.
• Applied histogram matching in YCbCr space to align the color distribution of generated images with real ones, reducing color-based confounds.
• Culled low-quality generated images using single-image FID scores, replacing them with higher-quality synthetic samples before retraining.
• Evaluated model performance using Fréchet Inception Distance (FID) and CLIP score to measure image quality and semantic alignment.
• Conducted control experiments with curated datasets to isolate the effects of image quality and color distribution on model collapse.

Figure 3: Shown is the Fréchet inception distance (FID) and contrastive language-image pre-training score (CLIP) as a function of the number of retraining iterations and the composition of the retraining dataset ranging from $100\%$ SD-generated faces and $0\%$ real faces to $0\%$ SD-generated and $

Experimental results

Research questions

• RQ1 What happens to the image quality and diversity of Stable Diffusion when retrained on even a small proportion of its own generated images?
• RQ2 Does the model collapse persist even after retraining on only real images, indicating irreversible damage?
• RQ3 Can preprocessing techniques such as color matching or quality curation prevent or delay model collapse?
• RQ4 To what extent does the poisoning effect extend beyond the specific text prompts used during retraining?
• RQ5 How generalizable is this collapse phenomenon across different generative AI models and training data compositions?

Key findings

• Stable Diffusion exhibits rapid model collapse when retrained on as little as 3% of its own generated images, with image quality degrading significantly by the fifth retraining iteration.
• Even after five additional retraining iterations on only real images, the model fails to recover, indicating persistent degradation and limited healing capacity.
• The FID score increases and CLIP score decreases across all poisoning ratios, with the most severe degradation observed at 100% self-generated data.
• Control experiments using color-matched and high-quality-curated retraining data still resulted in model collapse at the same rate, indicating that color and quality differences are not the root cause.
• The poisoning effect extends beyond the original prompts used in retraining, affecting model outputs for unrelated text prompts, suggesting broad distributional shift.
• The model shows an initial small improvement in image quality after the first retraining iteration, followed by consistent degradation, indicating transient optimization before collapse.