[Paper Review] Will Large-scale Generative Models Corrupt Future Datasets?

Ryuichiro Hataya, Han Bao+1 more|arXiv (Cornell University)|Nov 15, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper investigates the impact of large-scale generative images on future computer vision datasets by simulating contamination using StableDiffusion to create ImageNet- and COCO-scale synthetic datasets. Experiments show that generated images degrade downstream model performance across classification, captioning, and generation tasks, with performance dropping by up to 5.9% on ImageNet under high contamination, suggesting generative model outputs may compromise dataset quality and model robustness if not filtered.

ABSTRACT

Recently proposed large-scale text-to-image generative models such as DALL$\cdot$E 2, Midjourney, and StableDiffusion can generate high-quality and realistic images from users' prompts. Not limited to the research community, ordinary Internet users enjoy these generative models, and consequently, a tremendous amount of generated images have been shared on the Internet. Meanwhile, today's success of deep learning in the computer vision field owes a lot to images collected from the Internet. These trends lead us to a research question: " extbf{will such generated images impact the quality of future datasets and the performance of computer vision models positively or negatively?}" This paper empirically answers this question by simulating contamination. Namely, we generate ImageNet-scale and COCO-scale datasets using a state-of-the-art generative model and evaluate models trained with "contaminated" datasets on various tasks, including image classification and image generation. Throughout experiments, we conclude that generated images negatively affect downstream performance, while the significance depends on tasks and the amount of generated images. The generated datasets and the codes for experiments will be publicly released for future research. Generated datasets and source codes are available from \url{https://github.com/moskomule/dataset-contamination}.

Motivation & Objective

• To investigate whether large-scale generative images could contaminate future datasets collected from the internet, threatening the quality of computer vision models.
• To simulate dataset contamination by replacing real images in ImageNet and COCO with high-fidelity generated images from StableDiffusion.
• To evaluate the downstream performance of models trained on these contaminated datasets across multiple vision tasks.
• To analyze the root causes of performance degradation and suggest mitigation strategies for researchers and model publishers.
• To advocate for watermarking in generative models and filtering in dataset curation to preserve research integrity.

Proposed method

• Generated three million images using StableDiffusion, conditioned on ImageNet categories and COCO captions, to simulate large-scale dataset contamination.
• Created contaminated versions of ImageNet and COCO by replacing real images with generated ones at varying ratios (20%, 40%, 80%).
• Trained vision models from scratch on contaminated datasets for image classification, image captioning, and image generation tasks.
• Evaluated model performance using standard benchmarks: top-1 accuracy on ImageNet, BLEU and CIDEr scores on COCO captioning, and FID on image generation.
• Applied linear probing to MAE features to assess representation quality under contamination, measuring generalization and OOD robustness.
• Analyzed model robustness to out-of-distribution data and evaluated detectability of generated images using standard methods.

Figure 1: Schematic view of the problem. Some large-scale generative models are public, and many users are playing with them to share generated images on the Internet (top). Dataset collection heavily relies on images on the Internet, which may be contaminated by generated images (bottom). This pape

Experimental results

Research questions

• RQ1 How does contamination of ImageNet and COCO with generated images affect downstream image classification performance?
• RQ2 To what extent do generated images degrade performance in image captioning and image generation tasks?
• RQ3 Why do generated images degrade model performance despite high visual fidelity and coverage in synthetic evaluations?
• RQ4 Can self-supervised learning methods like MAE mitigate the negative effects of contaminated datasets?
• RQ5 How detectable are modern generative images, and what are the implications for dataset curation and model fairness?

Key findings

• Image classification accuracy on ImageNet dropped to 42.5% when 80% of the training data was replaced with generated images, a 1.4% decrease from the baseline.
• Performance degradation was most severe in image generation tasks, with FID scores increasing by 20.5 points when training on 80% generated images.
• Linear probing of MAE features showed that self-supervised representation learning could partially mitigate contamination effects, maintaining higher accuracy on corrupted features.
• Models trained on contaminated data exhibited reduced robustness to out-of-distribution inputs, indicating compromised generalization.
• Simple detection methods failed to identify generated images from state-of-the-art models like StableDiffusion, highlighting the need for advanced detection techniques.
• The study found that generative models capture fewer data distribution modes than real data, which contributes to performance degradation despite high visual quality.

Figure 2: Randomly selected examples from generated datasets, namely, SD-ImageNet (left) and SD-COCO (right). Images are at a single glance high quality and fidelity to prompts, i.e . , category names and captions, while details are unnatural, e.g . , a two-nose elephant.