[Paper Review] Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning

Weixin Liang, Yuhui Zhang+3 more|arXiv (Cornell University)|Mar 3, 2022

Multimodal Machine Learning Applications98 citations

TL;DR

The paper identifies and explains a modality gap in multi-modal contrastive representations, showing it arises from a cone effect due to initialization and is reinforced by contrastive learning; manipulating the gap can impact zero-shot performance and fairness.

ABSTRACT

We present modality gap, an intriguing geometric phenomenon of the representation space of multi-modal models. Specifically, we show that different data modalities (e.g. images and text) are embedded at arm's length in their shared representation in multi-modal models such as CLIP. Our systematic analysis demonstrates that this gap is caused by a combination of model initialization and contrastive learning optimization. In model initialization, we show empirically and theoretically that the representation of a common deep neural network is restricted to a narrow cone. As a consequence, in a multi-modal model with two encoders, the representations of the two modalities are clearly apart when the model is initialized. During optimization, contrastive learning keeps the different modalities separate by a certain distance, which is influenced by the temperature parameter in the loss function. Our experiments further demonstrate that varying the modality gap distance has a significant impact in improving the model's downstream zero-shot classification performance and fairness. Our code and data are available at https://modalitygap.readthedocs.io/

Motivation & Objective

• Demonstrate that a modality gap exists across multiple modalities and architectures.
• Explain the three-part mechanism behind the modality gap: cone effect from initialization, how random cones differ, and how contrastive learning preserves the gap.
• Show how varying the gap distance affects downstream zero-shot performance and fairness across tasks.

Proposed method

• Empirical visualization of embeddings (e.g., UMAP) to reveal a cone-shaped embedding space.
• Theoretical analysis of cone behavior across layers and the impact of non-linear activations on cosine similarity.
• Analysis of random initializations creating distinct embedding cones and their effect on modality gap.
• Loss landscape probing of CLIP to study how temperature and gap influence optimization.
• Embedding shift experiments to assess how closing or widening the gap affects contrastive loss.
• Controlled simulations and fine-tuning to examine temperature effects and gap manipulation.

Experimental results

Research questions

• RQ1 Does a modality gap exist across different modalities and architectures in multi-modal contrastive models?
• RQ2 What mechanisms (initialization cone effect, random cone variations, and contrastive loss dynamics) generate and maintain the gap?
• RQ3 How does changing the modality gap distance affect downstream zero-shot performance and fairness metrics?

Key findings

• Embedding spaces for images and texts lie in a narrow cone, even with random initialization or random noise inputs.
• Different random initializations produce distinct cones, explaining the modality gap at initialization in multi-encoder models.
• Deeper layers and non-linearities amplify cosine similarity, increasing cone narrowness (cone effect).
• Contrastive learning tends to preserve the modality gap; temperature influences the gap’s repulsive structure in the loss landscape.
• Manipulating the gap distance can improve zero-shot classification performance and fairness on several tasks, though effects vary by task and temperature.