[Paper Review] Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization

Bruno Korbar, Du Tran+1 more|arXiv (Cornell University)|Jun 30, 2018

Music and Audio Processing187 citations

TL;DR

A self-supervised two-stream framework learns audio and video representations by solving Audio-Visual Temporal Synchronization with curriculum-based hard negatives, yielding strong audio and video features for downstream tasks without manual labels.

ABSTRACT

There is a natural correlation between the visual and auditive elements of a video. In this work we leverage this connection to learn general and effective models for both audio and video analysis from self-supervised temporal synchronization. We demonstrate that a calibrated curriculum learning scheme, a careful choice of negative examples, and the use of a contrastive loss are critical ingredients to obtain powerful multi-sensory representations from models optimized to discern temporal synchronization of audio-video pairs. Without further finetuning, the resulting audio features achieve performance superior or comparable to the state-of-the-art on established audio classification benchmarks (DCASE2014 and ESC-50). At the same time, our visual subnet provides a very effective initialization to improve the accuracy of video-based action recognition models: compared to learning from scratch, our self-supervised pretraining yields a remarkable gain of +19.9% in action recognition accuracy on UCF101 and a boost of +17.7% on HMDB51.

Motivation & Objective

• Leverage the natural correlation between audio and visual signals in videos to learn general audiovisual representations without manual labeling.
• Train a two-stream network to determine if audio and video segments are temporally synchronized.
• Investigate the impact of negative sampling strategies and curriculum learning on representation quality for downstream tasks.
• Demonstrate that learned audio features match or exceed state-of-the-art on audio benchmarks and initialize video models for improved action recognition.

Proposed method

• Use a two-stream architecture with an audio subnetwork and a video subnetwork to assess synchronization.
• Train with a contrastive loss that minimizes distance for in-sync pairs and enforces a margin for out-of-sync pairs.
• Form positive examples from temporally aligned audio-video clips and negatives from either different videos (easy) or same video with temporal misalignment (hard).
• Apply curriculum learning by introducing hard negatives after initial learning on easy negatives, improving both AVTS and downstream tasks.
• Process video with MC x 3D/2D convolutions and audio with a VGG-like spectral feature extractor, then fuse late in the network for the AVTS task.
• Evaluate AVTS as a pretraining signal for action recognition and audio classification, including fine-tuning and zero-shot feature extraction scenarios.

Experimental results

Research questions

• RQ1 Can AVTS learn discriminative audio and video representations without manual labels?
• RQ2 How do negative sampling strategies (easy vs hard negatives) and curriculum learning affect AVTS performance and downstream tasks?
• RQ3 Do AVTS-learned features transfer to action recognition and audio classification tasks without labeled pretraining data?
• RQ4 What is the impact of pretraining with AVTS on larger unlabeled video/audio datasets compared to fully supervised pretraining?

Key findings

• Audio features learned via AVTS achieve competitive or superior results on DCASE2014 and ESC-50 without finetuning.
• Video subnet pretraining yields substantial gains in action recognition, e.g., large improvements on UCF101 and HMDB51 compared to learning from scratch.
• Curriculum learning with a mix of easy and hard negatives improves AVTS accuracy and downstream task performance more than single-stage training.
• AVTS pretraining on Kinetics provides substantial boosts for action recognition (e.g., MC3 model gains) and scales with larger unlabeled data (AudioSet).
• AVTS features generalize to audio benchmarks and support multi-modal action recognition outperforming related self-supervised methods.