[Paper Review] Self-Supervised Learning by Cross-Modal Audio-Video Clustering

Humam Alwassel, Dhruv Mahajan+4 more|arXiv (Cornell University)|Nov 28, 2019

Music and Audio Processing77 references251 citations

TL;DR

The paper introduces Cross-Modal Deep Clustering (XDC) for self-supervised learning from unlabeled videos by using clustering in one modality (audio or video) as the supervisory signal for the other, achieving state-of-the-art results and sometimes surpassing large-scale supervised pretraining.

ABSTRACT

Visual and audio modalities are highly correlated, yet they contain different information. Their strong correlation makes it possible to predict the semantics of one from the other with good accuracy. Their intrinsic differences make cross-modal prediction a potentially more rewarding pretext task for self-supervised learning of video and audio representations compared to within-modality learning. Based on this intuition, we propose Cross-Modal Deep Clustering (XDC), a novel self-supervised method that leverages unsupervised clustering in one modality (e.g., audio) as a supervisory signal for the other modality (e.g., video). This cross-modal supervision helps XDC utilize the semantic correlation and the differences between the two modalities. Our experiments show that XDC outperforms single-modality clustering and other multi-modal variants. XDC achieves state-of-the-art accuracy among self-supervised methods on multiple video and audio benchmarks. Most importantly, our video model pretrained on large-scale unlabeled data significantly outperforms the same model pretrained with full-supervision on ImageNet and Kinetics for action recognition on HMDB51 and UCF101. To the best of our knowledge, XDC is the first self-supervised learning method that outperforms large-scale fully-supervised pretraining for action recognition on the same architecture.

Motivation & Objective

• Motivation to reduce reliance on manually labeled video data for action understanding.
• Leverage the strong correlation yet complementary information between audio and video modalities.
• Propose cross-modal clustering frameworks that use one modality's pseudo-labels to train the other.
• Show that cross-modal self-supervision improves downstream action recognition and audio classification.

Proposed method

• Adapts DeepCluster-style self-supervision to a multi-modal setting with two encoders (visual E_v and audio E_a).
• Proposes three models: Multi-Head Deep Clustering (MDC), Concatenation Deep Clustering (CDC), and Cross-Modal Deep Clustering (XDC).
• MDC adds a second head per encoder supervised by the other modality’s cluster assignments.
• CDC clusters joint visual+audio features and uses those clusters as pseudo-labels for both encoders.
• XDC uses the other modality’s clusters as exclusive supervision for each encoder, enabling cross-modal self-supervision.
• Encoders produce modality-specific features which are clustered (k-means) to generate pseudo-labels that iteratively refine representations.

Experimental results

Research questions

• RQ1 How do multi-modal self-supervised clustering frameworks (MDC, CDC, XDC) compare to single-modality baselines?
• RQ2 What is the impact of the number of clusters (k) in k-means on XDC performance across datasets?
• RQ3 How does pretraining data type (curated vs uncurated) and size affect XDC’s transfer to downstream tasks?
• RQ4 Can XDC outperform fully supervised pretraining on standard action recognition and audio classification benchmarks?

Key findings

• All three multi-modal models outperform the single-modality DeepCluster baseline on downstream tasks.
• XDC consistently yields the best performance among the proposed models across evaluated datasets.
• XDC pretrained on large-scale unlabeled data can surpass fully supervised pretraining on Kinetics/ImageNet for action recognition in HMDB51/UCF101, marking a first in this setting.
• XDC pretrained on AudioSet or IG-Random/IG-Kinetics demonstrates strong transfer, with performance improving as pretraining data size increases.
• When used as a fixed feature extractor, XDC often outperforms several fully supervised models, and full finetuning with XDC remains competitive.