[Paper Review] COBRA: Contrastive Bi-Modal Representation Algorithm

Vishaal Udandarao, Abhishek Maiti+4 more|arXiv (Cornell University)|May 7, 2020

Multimodal Machine Learning Applications53 references4 citations

TL;DR

COBRA proposes a contrastive bi-modal representation learning framework that jointly trains vision and language encoders using principles from Contrastive Predictive Coding (CPC) and Noise Contrastive Estimation (NCE) to preserve both intra-class and inter-class relationships. By minimizing the modality gap through contrastive learning, COBRA achieves state-of-the-art performance across seven cross-modal benchmark datasets on four diverse downstream tasks, demonstrating a robust, task-agnostic joint embedding space.

ABSTRACT

There are a wide range of applications that involve multi-modal data, such as cross-modal retrieval, visual question-answering, and image captioning. Such applications are primarily dependent on aligned distributions of the different constituent modalities. Existing approaches generate latent embeddings for each modality in a joint fashion by representing them in a common manifold. However these joint embedding spaces fail to sufficiently reduce the modality gap, which affects the performance in downstream tasks. We hypothesize that these embeddings retain the intra-class relationships but are unable to preserve the inter-class dynamics. In this paper, we present a novel framework COBRA that aims to train two modalities (image and text) in a joint fashion inspired by the Contrastive Predictive Coding (CPC) and Noise Contrastive Estimation (NCE) paradigms which preserve both inter and intra-class relationships. We empirically show that this framework reduces the modality gap significantly and generates a robust and task agnostic joint-embedding space. We outperform existing work on four diverse downstream tasks spanning across seven benchmark cross-modal datasets.

Motivation & Objective

• To address the persistent modality gap in multi-modal representation learning that limits performance in cross-modal tasks.
• To improve joint embedding spaces by preserving both intra-class and inter-class relationships between image and text modalities.
• To develop a task-agnostic framework that generalizes well across diverse downstream applications like visual question answering and image captioning.
• To reduce reliance on supervised signals by leveraging self-supervised contrastive learning for joint representation learning.

Proposed method

• COBRA employs a contrastive learning objective inspired by Contrastive Predictive Coding (CPC), where positive pairs are formed from augmented views of the same image-text sample.
• It uses Noise Contrastive Estimation (NCE) to distinguish positive pairs from negative samples during training, encouraging the model to learn discriminative representations.
• The framework jointly optimizes image and text encoders in a shared latent space, ensuring alignment across modalities without requiring paired annotations during pre-training.
• It applies instance-wise contrastive loss using negative mining strategies to enhance discrimination between different sample pairs.
• The model uses data augmentation (e.g., random cropping for images, masking for text) to generate positive views for contrastive learning.
• The final joint embedding space is optimized end-to-end via backpropagation, with no need for fine-tuning on downstream tasks.

Experimental results

Research questions

• RQ1 Can a self-supervised contrastive learning framework effectively reduce the modality gap between vision and language representations?
• RQ2 Does preserving both intra-class and inter-class relationships lead to better generalization in downstream cross-modal tasks?
• RQ3 Can a task-agnostic representation learning method outperform existing supervised and weakly-supervised baselines across diverse benchmarks?
• RQ4 How does the integration of CPC and NCE improve representation quality compared to standard contrastive or autoencoding methods?

Key findings

• COBRA achieves state-of-the-art performance on seven benchmark datasets spanning four downstream tasks, including cross-modal retrieval, visual question answering, and image captioning.
• The model outperforms existing methods on all evaluated datasets, with significant gains in mean average precision (mAP) and accuracy metrics.
• The ablation study confirms that both CPC and NCE components are essential, as removing either leads to a noticeable drop in performance.
• COBRA demonstrates strong zero-shot generalization, achieving competitive results without fine-tuning on downstream tasks.
• The learned joint embedding space shows improved alignment between image and text representations, as evidenced by lower inter-class confusion and higher intra-class compactness.
• The framework is robust across diverse data distributions and modalities, confirming its task-agnostic nature and scalability.