[Paper Review] What to align in multimodal contrastive learning?

Benoit Dufumier, Javiera Castillo-Navarro+2 more|arXiv (Cornell University)|Sep 11, 2024

EFL/ESL Teaching and LearningArts and Humanities3 citations

TL;DR

This paper proposes CoMM, a contrastive multimodal self-supervised learning method that aligns augmented multimodal features to capture redundant, unique, and synergistic information beyond simple cross-modal redundancy. By maximizing mutual information between augmented multimodal representations, CoMM enables communication between modalities in a shared space, achieving state-of-the-art performance on six multimodal benchmarks while naturally modeling complex multimodal interactions.

ABSTRACT

Humans perceive the world through multisensory integration, blending the information of different modalities to adapt their behavior. Contrastive learning offers an appealing solution for multimodal self-supervised learning. Indeed, by considering each modality as a different view of the same entity, it learns to align features of different modalities in a shared representation space. However, this approach is intrinsically limited as it only learns shared or redundant information between modalities, while multimodal interactions can arise in other ways. In this work, we introduce CoMM, a Contrastive MultiModal learning strategy that enables the communication between modalities in a single multimodal space. Instead of imposing cross- or intra- modality constraints, we propose to align multimodal representations by maximizing the mutual information between augmented versions of these multimodal features. Our theoretical analysis shows that shared, synergistic and unique terms of information naturally emerge from this formulation, allowing us to estimate multimodal interactions beyond redundancy. We test CoMM both in a controlled and in a series of real-world settings: in the former, we demonstrate that CoMM effectively captures redundant, unique and synergistic information between modalities. In the latter, CoMM learns complex multimodal interactions and achieves state-of-the-art results on the six multimodal benchmarks.

Motivation & Objective

• To address the limitation of existing contrastive multimodal learning methods that only capture redundant information between modalities.
• To enable modeling of all types of multimodal interactions—redundancy, uniqueness, and synergy—within a unified self-supervised framework.
• To develop a method that allows true communication between modalities in a shared representation space, rather than forcing alignment between unimodal features.
• To provide a theoretically grounded approach that naturally emerges with shared, unique, and synergistic information terms from mutual information maximization.
• To achieve state-of-the-art performance on diverse multimodal benchmarks without relying on task-specific supervision or cross-modal contrastive constraints.

Proposed method

• CoMM uses a multimodal encoder to fuse inputs from multiple modalities into a single shared representation.
• It applies data augmentation to the multimodal features and maximizes the mutual information between augmented views of these features.
• The method is grounded in information theory, where mutual information maximization naturally gives rise to redundancy (R), uniqueness (U), and synergy (S) components.
• Unlike prior methods, CoMM does not impose cross- or intra-modal constraints; instead, it treats the multimodal representation as a unified entity for contrastive learning.
• The framework is compatible with existing vision and language backbones (e.g., CLIP, BLIP-2), with minimal computational overhead.
• The approach is inspired by the global workspace theory, modeling a shared representation space that integrates and broadcasts information across modalities.

Figure 1: a) We propose CoMM, a contrastive multimodal approach that allows the interplay of multiple modalities and learns multimodal interactions . Unlike previous multimodal models (Cross) that align cross-modal features, CoMM aligns multimodal features in a shared representation space. b) Multim

Experimental results

Research questions

• RQ1 Can contrastive learning be redefined to capture not just redundant, but also unique and synergistic information between modalities?
• RQ2 How can a self-supervised multimodal learning framework enable communication between modalities in a shared representation space without enforcing cross-modal alignment?
• RQ3 To what extent can mutual information maximization between augmented multimodal features yield a more comprehensive representation of multimodal interactions?
• RQ4 Does a contrastive objective based on multimodal features outperform traditional cross-modal contrastive methods in capturing complex, task-agnostic multimodal interactions?
• RQ5 Can such a method generalize across diverse multimodal benchmarks and achieve state-of-the-art performance without task-specific fine-tuning?

Key findings

• CoMM achieves state-of-the-art performance on six multimodal benchmarks, including MM-IMDb, Hateful Memes, and Vision&Touch, outperforming prior contrastive methods.
• In controlled experiments, CoMM successfully captures and disentangles redundant, unique, and synergistic information components between modalities.
• The method demonstrates that mutual information maximization between augmented multimodal features naturally gives rise to all three information terms (R, U, S) without explicit supervision.
• CoMM maintains strong performance with minimal computational overhead, adding only 30G FLOPs and 5ms latency over CLIP when using the same backbone.
• The model generalizes well across diverse tasks, including image-text classification, video action recognition, and robot manipulation, due to its ability to model complex multimodal interactions.
• Theoretical analysis confirms that CoMM's formulation aligns with cognitive principles of shared representation and information broadcasting, as in the global workspace theory.

Figure 2: CoMM’s model architecture. Inputs from different modalities $X=(X_{1},...,X_{n})$ are first encoded by modality-specific encoders. Modality-specific features are processed by latent converters to map them into sequences of embeddings which are concatenated and fused by a transformer block.