[Paper Review] Scaling Language-Image Pre-training via Masking

Yanghao Li, Haoqi Fan+3 more|arXiv (Cornell University)|Dec 1, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces Fast Language-Image Pre-training (FLIP), a method that accelerates CLIP training by randomly masking 50–75% of image patches during pre-training, enabling faster convergence and improved accuracy. FLIP achieves up to 3.7× faster training on 400M image-text pairs while outperforming CLIP baselines across diverse downstream tasks, demonstrating favorable trade-offs between speed and performance through efficient contrastive learning with sparse computation.

ABSTRACT

We present Fast Language-Image Pre-training (FLIP), a simple and more efficient method for training CLIP. Our method randomly masks out and removes a large portion of image patches during training. Masking allows us to learn from more image-text pairs given the same wall-clock time and contrast more samples per iteration with similar memory footprint. It leads to a favorable trade-off between accuracy and training time. In our experiments on 400 million image-text pairs, FLIP improves both accuracy and speed over the no-masking baseline. On a large diversity of downstream tasks, FLIP dominantly outperforms the CLIP counterparts trained on the same data. Facilitated by the speedup, we explore the scaling behavior of increasing the model size, data size, or training length, and report encouraging results and comparisons. We hope that our work will foster future research on scaling vision-language learning.

Motivation & Objective

• To address the high wall-clock training cost of large-scale vision-language pre-training, especially for CLIP-style models.
• To explore efficient scaling strategies for language-image pre-training without compromising model accuracy.
• To enable faster training with higher throughput by leveraging sparse computation through patch masking.
• To evaluate the impact of scaling model size, data size, and training length on FLIP's performance.
• To demonstrate that masking enables better trade-offs between training speed and accuracy than standard CLIP training.

Proposed method

• FLIP applies random masking to 50–75% of image patches during training, reducing computational load while preserving semantic supervision from text.
• The method uses a contrastive learning objective between masked image patches and their corresponding text descriptions, similar to CLIP.
• Masking allows for larger batch sizes (up to 4×) with minimal memory increase, enhancing gradient stability and accuracy.
• Training is optimized using LARS with cosine learning rate decay, weight decay, and layer-wise learning rate decay for better convergence.
• The model is fine-tuned end-to-end on downstream tasks such as image captioning, VQA, and zero-shot classification using standard fine-tuning protocols.
• A sequence-to-sequence transformer decoder is trained on top of the ViT encoder for image captioning, using teacher forcing and cross-entropy loss.

Experimental results

Research questions

• RQ1 Can patch masking during pre-training significantly reduce training time without degrading downstream performance?
• RQ2 How does masking affect the accuracy and throughput trade-off in contrastive language-image pre-training?
• RQ3 What are the scaling laws for model size, data size, and training length when using the FLIP method?
• RQ4 Does FLIP outperform CLIP and its variants when trained on the same data and evaluated on diverse downstream tasks?
• RQ5 Can masking enable more efficient exploration of large-scale pre-training due to reduced computational cost?

Key findings

• FLIP achieves a 3.7× speedup in wall-clock training time compared to CLIP on 256 TPU-v3 cores, reaching similar accuracy in less time.
• With the same number of training epochs, FLIP achieves higher zero-shot ImageNet-1K accuracy (86.1%) than CLIP (84.8%) on ViT-L/16.
• On a diverse set of downstream tasks, FLIP pre-trained on LAION-400M outperforms both OpenCLIP and a reproduced CLIP baseline across all benchmarks.
• Data scaling with FLIP shows performance gains at no additional training cost, indicating favorable data scaling behavior.
• Model scaling and data scaling both improve accuracy, with data scaling offering a particularly efficient path to performance gains.
• FLIP enables efficient fine-tuning for image captioning and VQA, with strong zero-shot generalization on COCO and NoCaps benchmarks.