[Paper Review] EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters

Quan Sun, Jinsheng Wang+5 more|arXiv (Cornell University)|Feb 6, 2024

Geomagnetism and Paleomagnetism Studies4 citations

TL;DR

EVA-CLIP-18B is a 18-billion-parameter contrastive language-image pretraining model that achieves state-of-the-art zero-shot accuracy of 80.7% across 27 image classification benchmarks, demonstrating consistent performance gains with model scaling despite using only 6 billion training samples from LAION-2B and COYO-700M. The model leverages EVA-style weak-to-strong visual scaling, achieving SOTA results in image, video, and 3D representation learning with publicly available weights.

ABSTRACT

Scaling up contrastive language-image pretraining (CLIP) is critical for empowering both vision and multimodal models. We present EVA-CLIP-18B, the largest and most powerful open-source CLIP model to date, with 18-billion parameters. With only 6-billion training samples seen, EVA-CLIP-18B achieves an exceptional 80.7% zero-shot top-1 accuracy averaged across 27 widely recognized image classification benchmarks, outperforming its forerunner EVA-CLIP (5-billion parameters) and other open-source CLIP models by a large margin. Remarkably, we observe a consistent performance improvement with the model size scaling of EVA-CLIP, despite maintaining a constant training dataset of 2-billion image-text pairs from LAION-2B and COYO-700M. This dataset is openly available and much smaller than the in-house datasets (e.g., DFN-5B, WebLI-10B) employed in other state-of-the-art CLIP models. EVA-CLIP-18B demonstrates the potential of EVA-style weak-to-strong visual model scaling. With our model weights made publicly available, we hope to facilitate future research in vision and multimodal foundation models.

Motivation & Objective

• To scale CLIP models to unprecedented size (18B parameters) while maintaining high performance on diverse vision benchmarks.
• To investigate whether consistent performance gains can be achieved through model scaling without increasing training data size.
• To validate the effectiveness of EVA-style weak-to-strong visual scaling in large-scale contrastive pretraining.
• To demonstrate that larger CLIP models can achieve superior zero-shot performance in image, video, and 3D representation learning.
• To release a powerful, open-source foundation model to accelerate research in vision and multimodal AI.

Proposed method

• The model is trained using a contrastive learning objective on a curated dataset of 2 billion image-text pairs from LAION-2B and COYO-700M, with no in-house data.
• EVA-CLIP-18B employs a vision transformer backbone with 18 billion parameters, trained with a global batch size of 108k on 360 A100 GPUs.
• The model uses a dual-stream architecture with separate vision and text encoders, jointly optimized via contrastive loss to align image and text embeddings.
• Training includes data augmentation and mixed-precision training to improve convergence and efficiency.
• The model is evaluated using zero-shot transfer protocols on 27 image classification, 4 video classification, and 2 retrieval benchmarks.
• The model is fine-tuned and evaluated at multiple resolutions (224×224, 336×336, 448×448) to study resolution effects.

Figure 1: Scaling behavior of EVA-CLIP with zero-shot classification performance averaged across 27 image classification benchmarks , compared with the current state-of-the-art and largest CLIP models (224px). The diameter of each circle demonstrates the forward GFLOPs $\times$ the number of trainin

Experimental results

Research questions

• RQ1 Does scaling up CLIP models to 18 billion parameters lead to consistent performance gains across diverse vision benchmarks, even with a fixed dataset size?
• RQ2 Can EVA-style weak-to-strong visual scaling principles be effectively applied to large-scale CLIP models to achieve SOTA performance?
• RQ3 How does the performance of EVA-CLIP-18B compare to other open-source and proprietary CLIP models in zero-shot image, video, and 3D classification?
• RQ4 What is the impact of increasing input resolution on zero-shot accuracy for EVA-CLIP-18B and its smaller counterparts?
• RQ5 Does incorporating video data during pretraining improve zero-shot video classification performance?

Key findings

• EVA-CLIP-18B achieves a new SOTA zero-shot top-1 accuracy of 80.7% averaged across 27 widely used image classification benchmarks.
• Despite using only 6 billion training samples, EVA-CLIP-18B outperforms its 5-billion-parameter predecessor and other open-source CLIP models by a significant margin.
• The model demonstrates consistent performance improvement with increasing model size, even when training data remains constant at 2 billion image-text pairs.
• In linear probing on ImageNet-1K, EVA-CLIP-18B achieves 88.9% top-1 accuracy, surpassing InternVL-C by 0.7 percentage points.
• In 3D representation learning, EVA-CLIP-18B as a teacher improves zero-shot performance on ModelNet40 and ScanObjectNN, achieving 87.6% and 65.3% top-1 accuracy respectively.
• Adding video data during pretraining improves zero-shot video classification by +0.7 (1-frame) and +0.8 (8-frame) on Kinetics and UCF-101 benchmarks.