[Paper Review] Reproducible scaling laws for contrastive language-image learning

Mehdi Cherti, Romain Beaumont+7 more|arXiv (Cornell University)|Dec 14, 2022

Multimodal Machine Learning Applications75 references29 citations

TL;DR

This paper shows power-law scaling laws for CLIP-like models trained on open public data (LAION-2B) across model size, data size, and samples seen, and compares OpenCLIP to OpenAI CLIP on zero-shot classification and retrieval tasks.

ABSTRACT

Scaling up neural networks has led to remarkable performance across a wide range of tasks. Moreover, performance often follows reliable scaling laws as a function of training set size, model size, and compute, which offers valuable guidance as large-scale experiments are becoming increasingly expensive. However, previous work on scaling laws has primarily used private data \& models or focused on uni-modal language or vision learning. To address these limitations, we investigate scaling laws for contrastive language-image pre-training (CLIP) with the public LAION dataset and the open-source OpenCLIP repository. Our large-scale experiments involve models trained on up to two billion image-text pairs and identify power law scaling for multiple downstream tasks including zero-shot classification, retrieval, linear probing, and end-to-end fine-tuning. We find that the training distribution plays a key role in scaling laws as the OpenAI and OpenCLIP models exhibit different scaling behavior despite identical model architectures and similar training recipes. We open-source our evaluation workflow and all models, including the largest public CLIP models, to ensure reproducibility and make scaling laws research more accessible. Source code and instructions to reproduce this study will be available at https://github.com/LAION-AI/scaling-laws-openclip

Motivation & Objective

• Investigate how scaling model size, data size, and samples seen affects downstream CLIP performance.
• Assess whether scaling laws hold for multi-modal learning with public data and open-source code.
• Compare scaling behavior of OpenCLIP (LAION) with OpenAI CLIP (WIT) across tasks.

Proposed method

• Train CLIP models using OpenCLIP across multiple scales: ViT-B/32, B/16, L/14, H/14, g/14.
• Use LAION-80M, LAION-400M, and LAION-2B data subsets with 3B, 13B, and 34B samples seen.
• Evaluate with zero-shot classification, image/text retrieval, linear probing, and fine-tuning on diverse downstream tasks.
• Fit power laws on Pareto frontiers of models to relate performance to total training compute, data, and samples seen.
• Open-source evaluation workflow and models are released for reproducibility.

Experimental results

Research questions

• RQ1 Do scaling laws in the form of power laws hold for contrastive language-image pre-training (CLIP) when trained on public datasets?
• RQ2 How do model size, data size, and number of samples seen interact to affect zero-shot classification and retrieval performance?
• RQ3 Do OpenCLIP models trained on LAION data exhibit different scaling behavior compared to OpenAI CLIP models trained on WIT, and why?
• RQ4 How do scaling trends transfer to robustness benchmarks and linear probing/fine-tuning scenarios?

Key findings

• Zero-shot performance (classification and retrieval) follows power-law scaling with model/data/samples seen.
• OpenCLIP on LAION-2B shows stronger scaling for retrieval tasks, while OpenAI CLIP on WIT shows stronger scaling for zero-shot ImageNet classification.
• Bottleneck effects observed: some scaling dimensions limit gains unless others are increased (e.g., data vs samples seen).
• Linear probes and fine-tuning benefits persist with scale, showing improved performance with larger data/model and more samples seen.
• Predictions using the scaling laws suggest substantial gains at larger scales and comparable robustness improvements with scale.