[Paper Review] Identifiability Results for Multimodal Contrastive Learning

Imant Daunhawer, Alice Bizeul+3 more|arXiv (Cornell University)|Mar 16, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper establishes theoretical identifiability results for multimodal contrastive learning by modeling distinct generative mechanisms per modality with modality-specific latent variables. It proves that contrastive learning can block-identify shared content factors even under nontrivial causal and statistical dependencies between latent components, validated through controlled simulations and a complex image/text dataset with high-dimensional observations.

ABSTRACT

Contrastive learning is a cornerstone underlying recent progress in multi-view and multimodal learning, e.g., in representation learning with image/caption pairs. While its effectiveness is not yet fully understood, a line of recent work reveals that contrastive learning can invert the data generating process and recover ground truth latent factors shared between views. In this work, we present new identifiability results for multimodal contrastive learning, showing that it is possible to recover shared factors in a more general setup than the multi-view setting studied previously. Specifically, we distinguish between the multi-view setting with one generative mechanism (e.g., multiple cameras of the same type) and the multimodal setting that is characterized by distinct mechanisms (e.g., cameras and microphones). Our work generalizes previous identifiability results by redefining the generative process in terms of distinct mechanisms with modality-specific latent variables. We prove that contrastive learning can block-identify latent factors shared between modalities, even when there are nontrivial dependencies between factors. We empirically verify our identifiability results with numerical simulations and corroborate our findings on a complex multimodal dataset of image/text pairs. Zooming out, our work provides a theoretical basis for multimodal representation learning and explains in which settings multimodal contrastive learning can be effective in practice.

Motivation & Objective

• To formalize a more general generative model for multimodal data that distinguishes between modality-specific and shared latent factors.
• To extend prior identifiability results from multi-view to multimodal settings with distinct generative mechanisms.
• To prove that contrastive learning can recover shared latent factors up to block-wise indeterminacies under nontrivial dependencies between factors.
• To empirically validate theoretical findings using numerical simulations and a complex image/text dataset with disentangled factors.

Proposed method

• Formalize a multimodal generative process using modality-specific mixing functions and shared latent factors, distinguishing content, style, and modality-specific components.
• Introduce a latent variable model where each modality has its own generative mechanism with modality-specific latent variables.
• Prove that contrastive learning via the InfoNCE objective can block-identify shared latent factors even when dependencies exist between components.
• Use kernel ridge regression to evaluate representation quality by predicting ground truth factors from learned embeddings.
• Design a synthetic multimodal dataset (Multimodal3DIdent) with controllable factors including causal dependencies between content and style.
• Train contrastive models on image pairs and image-text pairs, measuring performance via R2 scores and classification accuracy for continuous and discrete factors.

Experimental results

Research questions

• RQ1 Can contrastive learning identify shared latent factors in a multimodal setting with distinct generative mechanisms per modality?
• RQ2 Under what conditions can contrastive learning recover shared factors when there are nontrivial dependencies between latent components?
• RQ3 How does modality-specific variation affect the identifiability of shared content factors in contrastive learning?
• RQ4 To what extent do learned representations encode content factors versus style or modality-specific factors in high-dimensional, realistic settings?
• RQ5 Does the theoretical identifiability hold in practice on complex multimodal datasets with continuous and discrete factors?

Key findings

• Contrastive learning successfully block-identifies content factors (e.g., object position) in image pairs when sufficient encoding capacity is available, achieving high R2 scores.
• Style and modality-specific factors are largely discarded by the model, even with increased encoding size, indicating robust disentanglement.
• In settings with causal dependencies between style and content, some style information is recovered, consistent with theoretical expectations.
• The model achieves strong performance in predicting ground truth content factors from representations, with R2 scores approaching 1.0 for object position and shape in the Multimodal3DIdent dataset.
• Empirical results on image/text pairs show that content factors are encoded effectively, while discrete and continuous style factors are not preserved, supporting theoretical claims.
• The findings hold across different encoding sizes and are robust across three random seeds, with consistent performance bands in all experiments.