[Paper Review] Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm

Yangguang Li, Feng Liang+6 more|arXiv (Cornell University)|Oct 11, 2021

Multimodal Machine Learning ApplicationsComputer Science39 references127 citations

TL;DR

DeCLIP introduces self-supervision, multi-view supervision, and nearest-neighbor supervision to improve data efficiency in contrastive language-image pre-training, achieving strong zero-shot and transfer performance with significantly less data than CLIP.

ABSTRACT

Recently, large-scale Contrastive Language-Image Pre-training (CLIP) has attracted unprecedented attention for its impressive zero-shot recognition ability and excellent transferability to downstream tasks. However, CLIP is quite data-hungry and requires 400M image-text pairs for pre-training, thereby restricting its adoption. This work proposes a novel training paradigm, Data efficient CLIP (DeCLIP), to alleviate this limitation. We demonstrate that by carefully utilizing the widespread supervision among the image-text pairs, our De-CLIP can learn generic visual features more efficiently. Instead of using the single image-text contrastive supervision, we fully exploit data potential through the use of (1) self-supervision within each modality; (2) multi-view supervision across modalities; (3) nearest-neighbor supervision from other similar pairs. Benefiting from intrinsic supervision, our DeCLIP-ResNet50 can achieve 60.4% zero-shot top1 accuracy on ImageNet, which is 0.8% above the CLIP-ResNet50 while using 7.1 x fewer data. Our DeCLIP-ResNet50 outperforms its counterpart in 8 out of 11 visual datasets when transferred to downstream tasks. Moreover, Scaling up the model and computing also works well in our framework.Our code, dataset and models are released at: https://github.com/Sense-GVT/DeCLIP

Motivation & Objective

• Motivate data-efficient visual feature learning from image-text pairs without relying on massive datasets.
• Exploit intrinsic supervision within each modality and across modalities to learn robust representations.
• Introduce nearest-neighbor supervision to leverage similar captions across pairs.
• Demonstrate data efficiency and transferability across multiple architectures and datasets.

Proposed method

• Build on the CLIP framework with a two-tower image and text encoder setup.
• Add Self-Supervision within each modality using SimSiam for images and MLM for text.
• Introduce Multi-View Supervision by contrasting 2x2 image-text pairs from augmented views.
• Propose Nearest-Neighbor Supervision by sampling nearest text embeddings as additional supervision via a FIFO embedding queue.
• Combine losses into L_DeCLIP = (1-α-β-γ)L_CLIP + αL_ISS + αL_TSS + βL_MVS + γL_NNS.

Experimental results

Research questions

• RQ1 Can intrinsic supervision within multimodal data improve data efficiency in language-image pre-training?
• RQ2 How do self-supervision, multi-view, and nearest-neighbor signals contribute to zero-shot and transfer performance?
• RQ3 What is the data efficiency and scalability of DeCLIP across different encoder architectures and dataset sizes?
• RQ4 Does DeCLIP maintain competitive or superior performance on downstream tasks with reduced pre-training data?

Key findings

• DeCLIP achieves 60.4% zero-shot top-1 on ImageNet with 88M data, 0.8% above CLIP-ResNet50 using 7.1x fewer data.
• With the same data budget (88M), DeCLIP-ResNet50/ViT-B32 reach 62.5% and 66.2% zero-shot accuracy, respectively, outperforming their CLIP counterparts.
• Scaling to larger models (RegNetY-64GF + BERT) yields 73.7% zero-shot accuracy with 88M data, comparable to CLIP-R50×64 while using fewer resources.
• DeCLIP improves transfer performance on 8 of 11 downstream datasets (average gain ~0.8%) compared to CLIP.
• Ablation shows self-supervision, multi-view, and nearest-neighbor signals each contribute to performance gains, with NN supervision adding notable improvements.