[Paper Review] Distilling Audio-Visual Knowledge by Compositional Contrastive Learning

Yanbei Chen, Yongqin Xian+3 more|arXiv (Cornell University)|Apr 22, 2021

Multimodal Machine Learning Applications64 references4 citations

TL;DR

This paper proposes Compositional Contrastive Learning (CCL), a novel knowledge distillation framework that transfers heterogeneous audio, image, and video representations to improve video representation learning by learning compositional embeddings to bridge cross-modal semantic gaps. CCL outperforms state-of-the-art methods on UCF101, ActivityNet, and VGGSound, achieving up to 4.5% accuracy gain in video recognition through joint contrastive learning in a shared latent space with class-aware supervision.

ABSTRACT

Having access to multi-modal cues (e.g. vision and audio) empowers some cognitive tasks to be done faster compared to learning from a single modality. In this work, we propose to transfer knowledge across heterogeneous modalities, even though these data modalities may not be semantically correlated. Rather than directly aligning the representations of different modalities, we compose audio, image, and video representations across modalities to uncover richer multi-modal knowledge. Our main idea is to learn a compositional embedding that closes the cross-modal semantic gap and captures the task-relevant semantics, which facilitates pulling together representations across modalities by compositional contrastive learning. We establish a new, comprehensive multi-modal distillation benchmark on three video datasets: UCF101, ActivityNet, and VGGSound. Moreover, we demonstrate that our model significantly outperforms a variety of existing knowledge distillation methods in transferring audio-visual knowledge to improve video representation learning. Code is released here: https://github.com/yanbeic/CCL.

Motivation & Objective

• To address the challenge of transferring knowledge from heterogeneous, uncorrelated audio and visual modalities to video representation learning in unconstrained settings.
• To close the cross-modal semantic gap between unimodal teachers (audio, image, video) and a video student network, especially when modalities are not temporally or semantically aligned.
• To develop a generic, unified framework for multi-modal distillation that works across diverse datasets and tasks without requiring modality-specific alignment.
• To establish a comprehensive benchmark for multi-modal distillation on UCF101, ActivityNet, and VGGSound, enabling fair comparison of distillation methods.
• To demonstrate that jointly contrasting representations across modalities using class labels and compositional embeddings leads to more informative knowledge transfer than standard contrastive or distillation baselines.

Proposed method

• A learnable compositional embedding is introduced to map representations from different modalities into a shared latent space, effectively reducing the cross-modal semantic gap.
• The method employs a multi-class contrastive loss (NCE) that contrasts positive and negative samples across modalities, using class labels to guide alignment.
• A Jensen-Shannon divergence (JSD) loss is used to align the prediction distributions of the student and teacher networks, ensuring knowledge transfer in the output space.
• The framework jointly optimizes the compositional contrastive loss and JSD loss, enabling joint alignment in both feature and prediction spaces.
• The approach is generic and modular, allowing plug-in of pre-trained teacher networks from any modality (e.g., ImageNet for images, LibriSpeech for audio) without architectural constraints.
• The method is applied to both video recognition and retrieval tasks, with extensive ablation and analysis on three benchmark datasets.

Experimental results

Research questions

• RQ1 Can compositional contrastive learning effectively transfer knowledge from heterogeneous, uncorrelated audio and visual modalities to video representation learning?
• RQ2 How does introducing class labels into the contrastive learning objective improve multi-modal knowledge distillation compared to instance-level contrastive learning?
• RQ3 To what extent do joint constraints in feature and prediction space (via NCE and JSD losses) synergistically enhance knowledge transfer?
• RQ4 How does the model perform under realistic conditions where audio and video are not semantically or temporally aligned?
• RQ5 What is the impact of compositional embeddings in closing the cross-modal semantic gap when modalities are not correlated?

Key findings

• CCL achieves a 4.5% absolute accuracy gain over the baseline using instance-level contrastive loss (64.9% vs. 60.4%) on the UCF101 (A) setup, demonstrating the benefit of class-aware contrastive learning.
• Removing either the NCE or JSD loss leads to significant performance drops—1.8% and 1.7% respectively—indicating that both losses are complementary and essential for effective distillation.
• The model achieves a 3.7% accuracy drop (70.0% to 66.3%) when the NCE loss is removed on the (AI) setup, confirming its critical role in feature-level alignment.
• Qualitative t-SNE visualizations show that CCL learns discriminative, class-separated video representations, with training and test set embeddings forming consistent, clustered manifolds.
• In k-NN retrieval, CCL successfully retrieves videos of the same or similar action classes, such as retrieving 'ice dancing' and 'apply eye makeup' videos for corresponding queries.
• Visual analysis confirms that CCL generalizes well even when audio and video are not semantically correlated (e.g., music with unrelated actions), thanks to the compositional embedding mechanism.