[论文解读] LAION-5B: An open large-scale dataset for training next generation image-text models

Christoph Schuhmann, Romain Beaumont+14 more|arXiv (Cornell University)|Oct 16, 2022

Natural Language Processing Techniques被引用 1,035

一句话总结

LAION-5B 提供一个公开的大规模数据集，包含 5.85B 的图文对，经 CLIP 过滤，使得在大规模上复现和微调 CLIP 类模型与生成模型成为可能。

ABSTRACT

Groundbreaking language-vision architectures like CLIP and DALL-E proved the utility of training on large amounts of noisy image-text data, without relying on expensive accurate labels used in standard vision unimodal supervised learning. The resulting models showed capabilities of strong text-guided image generation and transfer to downstream tasks, while performing remarkably at zero-shot classification with noteworthy out-of-distribution robustness. Since then, large-scale language-vision models like ALIGN, BASIC, GLIDE, Flamingo and Imagen made further improvements. Studying the training and capabilities of such models requires datasets containing billions of image-text pairs. Until now, no datasets of this size have been made openly available for the broader research community. To address this problem and democratize research on large-scale multi-modal models, we present LAION-5B - a dataset consisting of 5.85 billion CLIP-filtered image-text pairs, of which 2.32B contain English language. We show successful replication and fine-tuning of foundational models like CLIP, GLIDE and Stable Diffusion using the dataset, and discuss further experiments enabled with an openly available dataset of this scale. Additionally we provide several nearest neighbor indices, an improved web-interface for dataset exploration and subset generation, and detection scores for watermark, NSFW, and toxic content detection. Announcement page https://laion.ai/laion-5b-a-new-era-of-open-large-scale-multi-modal-datasets/

研究动机与目标

• 通过公开的大规模数据集实现多模态研究的普及化
• 使在多样的下游任务上能够复现实验并对 CLIP 类模型进行基准测试
• 提供工具和安全措施（NSFW、水印、多语言）以研究偏见和内容安全性
• 探索数据集规模对零样本与鲁棒性性能的影响
• 讨论大规模未经过筛选数据集的伦理考量与局限性

提出的方法

• 使用三阶段管线从 Common Crawl 汇集数据（网页筛选、图文对下载、内容过滤）
• 使用 ViT-B/32 的 CLIP 对英文及多语言变体基于余弦相似度对图文对进行筛选
• 用 CLD3 对文本语言进行分类，并应用语言感知的筛选和元数据标注
• 提供最近邻搜索接口以及子集提取（如 LAION-2B-en、LAION-Aesthetic）
• 发布包含安全标记（NSFW、水印）和 Parquet 元数据的开源工具与数据集
• 在 LAION-400M 和 LAION-2B-en 上进行 CLIP 复现实验以验证零样本和鲁棒性性能

实验结果

研究问题

• RQ1 公开的大规模图文数据集是否能够达到行业巨头所使用的大型封闭数据集在零样本和鲁棒性上的性能？
• RQ2 在开放的 CLIP 类训练中，数据与模型规模的增加如何影响下游的零样本准确率和VTAB 类基准？
• RQ3 在未经过筛选的网络数据上训练多模态模型时有哪些影响、偏见和安全考虑？
• RQ4 基于 CLIP 的过滤在十亿级数据规模下进行质量控制的有效性如何，且此类过滤的极限在哪里？

主要发现

• 在 LAION-400M 和 2B-en 上训练的 CLIP 模型在零样本 ImageNet 和鲁棒性性能方面，与在私有 WIT 数据上训练的 OpenAI CLIP 相当。
• Scaling to LAION-2B-en over LAION-400M improves performance across multiple datasets and prompts, given sufficient compute.
• LAION-5B enables successful replication and fine-tuning of generation models like GLIDE, demonstrating transfer to generation tasks.
• The authors provide multiple subsets and tools (e.g., nearest-neighbor indices, CLIP retrieval UI) to facilitate research and dataset exploration.
• The dataset is not a finished product and should be used for academic research with careful analysis of biases and safety implications.
• Overlaps with downstream test sets and the quality/noise of alt-text are important considerations in evaluating generalization.