[Paper Review] Active Contrastive Learning of Audio-Visual Video Representations

Shuang Ma, Zhaoyang Zeng+2 more|arXiv (Cornell University)|Aug 31, 2020

Speech and Audio Processing80 references46 citations

TL;DR

CM-ACC introduces an actively sampled cross-modal contrastive framework for audio-visual video representations, yielding strong improvements on standard benchmarks and mitigating redundancy in large dictionaries.

ABSTRACT

Contrastive learning has been shown to produce generalizable representations of audio and visual data by maximizing the lower bound on the mutual information (MI) between different views of an instance. However, obtaining a tight lower bound requires a sample size exponential in MI and thus a large set of negative samples. We can incorporate more samples by building a large queue-based dictionary, but there are theoretical limits to performance improvements even with a large number of negative samples. We hypothesize that extit{random negative sampling} leads to a highly redundant dictionary that results in suboptimal representations for downstream tasks. In this paper, we propose an active contrastive learning approach that builds an extit{actively sampled} dictionary with diverse and informative items, which improves the quality of negative samples and improves performances on tasks where there is high mutual information in the data, e.g., video classification. Our model achieves state-of-the-art performance on challenging audio and visual downstream benchmarks including UCF101, HMDB51 and ESC50.\footnote{Code is available at: \url{https://github.com/yunyikristy/CM-ACC}}

Motivation & Objective

• Motivate learning robust audio-visual video representations via self-supervised contrastive learning.
• Address redundancy in large dictionaries and ineffective negatives by proposing active sampling.
• Develop a cross-modal contrastive framework that leverages audio-visual correspondence.
• Demonstrate improved downstream performance on action and sound classification tasks.

Proposed method

• Extend MoCo-style contrastive learning to cross-modal audio-visual video data.
• Construct actively sampled dictionaries of negative samples using gradient-based uncertainty and k-means++ initialization for diversity.
• Compute cross-modal contrastive loss with query encoders and momentum-updated key encoders across modalities.
• Use pseudo-labels to guide uncertainty estimation and select diverse, informative negatives.
• Incorporate an additional FC layer to allow cross-modal gradient flow and improve stability.
• Evaluate on downstream tasks after pretraining on AudioSet and Kinetics datasets.

Experimental results

Research questions

• RQ1 Can active, uncertainty-guided negative sampling improve contrastive learning for high-MI audio-visual video data compared to random negatives?
• RQ2 Does cross-modal contrastive learning with actively sampled dictionaries yield better representations for downstream video action and audio classification tasks?
• RQ3 How does gradient-based negative sampling compare to feature-based sampling in terms of diversity and downstream performance?
• RQ4 What is the impact of cross-modal gradient flow on training stability and representation quality?

Key findings

• CM-ACC achieves state-of-the-art classification on UCF101, HMDB51, and ESC50 when pretrained on AudioSet and Kinetics data.
• Active sampling yields more diverse negatives across semantic categories than random sampling, covering a higher percentage of categories.
• Gradient-based embedding for negative sampling provides stronger downstream gains than using feature embeddings alone.
• CM-ACC outperforms random-sampling MoCo baselines by sizeable margins across multiple benchmarks (e.g., up to +6.2 on ESC50 in reported comparisons).
• Cross-modal training with gradient-flow through an auxiliary FC layer enhances stability and performance compared to versions without cross-modal flow.