[Paper Review] LAION-5B: An open large-scale dataset for training next generation image-text models

Christoph Schuhmann, Romain Beaumont+14 more|arXiv (Cornell University)|Oct 16, 2022

Natural Language Processing Techniques1,035 citations

TL;DR

LAION-5B provides a public open dataset with 5.85B image-text pairs filtered by CLIP, enabling replication and fine-tuning of CLIP-like and generation models at scale.

ABSTRACT

Groundbreaking language-vision architectures like CLIP and DALL-E proved the utility of training on large amounts of noisy image-text data, without relying on expensive accurate labels used in standard vision unimodal supervised learning. The resulting models showed capabilities of strong text-guided image generation and transfer to downstream tasks, while performing remarkably at zero-shot classification with noteworthy out-of-distribution robustness. Since then, large-scale language-vision models like ALIGN, BASIC, GLIDE, Flamingo and Imagen made further improvements. Studying the training and capabilities of such models requires datasets containing billions of image-text pairs. Until now, no datasets of this size have been made openly available for the broader research community. To address this problem and democratize research on large-scale multi-modal models, we present LAION-5B - a dataset consisting of 5.85 billion CLIP-filtered image-text pairs, of which 2.32B contain English language. We show successful replication and fine-tuning of foundational models like CLIP, GLIDE and Stable Diffusion using the dataset, and discuss further experiments enabled with an openly available dataset of this scale. Additionally we provide several nearest neighbor indices, an improved web-interface for dataset exploration and subset generation, and detection scores for watermark, NSFW, and toxic content detection. Announcement page https://laion.ai/laion-5b-a-new-era-of-open-large-scale-multi-modal-datasets/

Motivation & Objective

• democratize multimodal research by releasing a public dataset at scale
• enable replication and benchmarking of CLIP-like models on diverse downstream tasks
• provide tools and safeguards (NSFW, watermark, multilingual) to study biases and content safety
• explore the impact of dataset scale on zero-shot and robustness performance
• discuss ethical considerations and limitations of large uncurated datasets

Proposed method

• assemble data from Common Crawl using a three-stage pipeline (webpage filtering, image-text pair downloading, content filtering)
• filter image-text pairs with CLIP-based cosine similarity using ViT-B/32 CLIP for English and multilingual variants
• categorize text language with CLD3 and apply language-aware filtering and metadata tagging
• provide nearest-neighbor search interface and extraction of subsets (e.g., LAION-2B-en, LAION-Aesthetic)
• release open-source tooling and datasets with safety tagging (NSFW, watermark) and Parquet metadata
• conduct CLIP-reproduction experiments on LAION-400M and LAION-2B-en to validate zero-shot and robustness performance

Experimental results

Research questions

• RQ1 Can a public, large-scale image-text dataset match the zero-shot and robustness performance of large closed datasets used by industry players?
• RQ2 How does increasing data and model scale on open CLIP-like training affect downstream zero-shot accuracy and VTAB-like benchmarks?
• RQ3 What are the implications, biases, and safety considerations when training multimodal models on uncurated web data?
• RQ4 How effective is CLIP-based filtering at quality-controlling data at billion-scale, and what are the limits of such filtering?

Key findings

• CLIP models trained on LAION-400M and 2B-en achieve competitive zero-shot ImageNet and robustness performance compared to OpenAI’s CLIP trained on private WIT data.
• Scaling to LAION-2B-en over LAION-400M improves performance across multiple datasets and prompts, given sufficient compute.
• LAION-5B enables successful replication and fine-tuning of generation models like GLIDE, demonstrating transfer to generation tasks.
• The authors provide multiple subsets and tools (e.g., nearest-neighbor indices, CLIP retrieval UI) to facilitate research and dataset exploration.
• The dataset is not a finished product and should be used for academic research with careful analysis of biases and safety implications.
• Overlaps with downstream test sets and the quality/noise of alt-text are important considerations in evaluating generalization.