[Paper Review] Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-training

Haoxuan You, Luowei Zhou+6 more|arXiv (Cornell University)|Jul 26, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes Modality-Shared Contrastive Language-Image Pre-training (MS-CLIP), a unified transformer-based architecture that shares most parameters between vision and language encoders during contrastive pre-training. By sharing self-attention and feed-forward weights across modalities while using lightweight modality-specific modules, MS-CLIP achieves 13% relative improvement in zero-shot ImageNet accuracy over vanilla CLIP with fewer parameters, while enabling better alignment of semantic structures across modalities.

ABSTRACT

Large-scale multi-modal contrastive pre-training has demonstrated great utility to learn transferable features for a range of downstream tasks by mapping multiple modalities into a shared embedding space. Typically, this has employed separate encoders for each modality. However, recent work suggests that transformers can support learning across multiple modalities and allow knowledge sharing. Inspired by this, we investigate a variety of Modality-Shared Contrastive Language-Image Pre-training (MS-CLIP) frameworks. More specifically, we question how many parameters of a transformer model can be shared across modalities during contrastive pre-training, and rigorously examine architectural design choices that position the proportion of parameters shared along a spectrum. In studied conditions, we observe that a mostly unified encoder for vision and language signals outperforms all other variations that separate more parameters. Additionally, we find that light-weight modality-specific parallel modules further improve performance. Experimental results show that the proposed MS-CLIP approach outperforms vanilla CLIP by up to 13\% relative in zero-shot ImageNet classification (pre-trained on YFCC-100M), while simultaneously supporting a reduction of parameters. In addition, our approach outperforms vanilla CLIP by 1.6 points in linear probing on a collection of 24 downstream vision tasks. Furthermore, we discover that sharing parameters leads to semantic concepts from different modalities being encoded more closely in the embedding space, facilitating the transferring of common semantic structure (e.g., attention patterns) from language to vision. Code is available at \href{https://github.com/Hxyou/MSCLIP}{URL}.

Motivation & Objective

• To investigate the feasibility and performance impact of sharing transformer parameters between vision and language encoders in contrastive pre-training.
• To determine which components (e.g., attention, normalization, embeddings) should be shared or kept modality-specific for optimal performance.
• To evaluate whether lightweight modality-specific modules can improve performance by balancing cross-modal generalization and modality-specific specialization.
• To analyze whether parameter sharing leads to more aligned semantic structures (e.g., attention patterns) across modalities.

Proposed method

• Proposes a modality-shared contrastive pre-training framework (MS-CLIP) where most transformer components—self-attention and feed-forward networks—are shared between vision and language encoders.
• Identifies that input embeddings, layer normalization, and output projection should be modality-specific, while all transformer layers can be shared.
• Introduces two auxiliary designs: Early Specialization (replacing the first transformer block with modality-specific CNNs and a transformer layer) and an Efficient Parallel Branch (adding a lightweight multi-scale CNN for vision via depth-wise conv adaptors).
• Employs contrastive loss on large-scale image-caption datasets (YFCC-100M and Laion-400M) to pre-train the unified model.
• Measures cross-modal semantic alignment using a Common Semantic Structure (CSC) distance metric based on attention pattern similarity between visual and textual concepts.
• Visualizes shared attention heads to demonstrate co-reference learning across modalities, such as attending to 'cat' in image and 'cats' in text.

Experimental results

Research questions

• RQ1 Which sub-modules within each transformer layer should be shared between vision and language encoders to maximize performance?
• RQ2 What is the optimal degree of layer sharing across vision and language encoders in a contrastive pre-training setting?
• RQ3 Can lightweight modality-specific components improve performance by accommodating modality-specific characteristics without breaking parameter sharing?
• RQ4 To what extent does parameter sharing align semantic structures (e.g., attention patterns) across vision and language modalities?

Key findings

• MS-CLIP outperforms vanilla CLIP by 13% relative in zero-shot ImageNet classification when pre-trained on YFCC-100M, despite having fewer parameters.
• The model achieves a 1.6-point improvement in linear probing accuracy across 24 downstream vision tasks compared to vanilla CLIP.
• Sharing all transformer layers, including self-attention and feed-forward weights, yields better performance than selective layer sharing or NAS-based strategies.
• The proposed Early Specialization and Auxiliary Parallel Branch modules further improve performance by enabling early modality-specific abstraction and enhanced spatial modeling.
• The Common Semantic Structure (CSC) distance is significantly reduced in MS-CLIP, indicating that shared parameters promote more similar attention patterns across modalities.
• Visualization confirms that shared attention heads learn co-reference relationships, such as attending to 'cat' in image and 'cats' in text, demonstrating cross-modal alignment.