[Paper Review] Co-Training of Audio and Video Representations from Self-Supervised Temporal Synchronization

Bruno Korbar, Du Tran+1 more|arXiv (Cornell University)|Jun 30, 2018

Music and Audio Processing27 references52 citations

TL;DR

This paper proposes a self-supervised method that learns powerful audio and video representations by training models to recognize temporal synchronization between audio and video streams. Using contrastive learning with a curriculum strategy and careful negative sampling, the approach achieves state-of-the-art performance on audio benchmarks and boosts action recognition accuracy by +16.7% on UCF101 and +13.0% on HMDB51 when fine-tuned from self-supervised pretraining.

ABSTRACT

There is a natural correlation between the visual and auditive elements of a video. In this work we leverage this connection to learn general and effective features for both audio and video analysis from self-supervised temporal synchronization. We demonstrate that a calibrated curriculum learning scheme, a careful choice of negative examples, and the use of a contrastive loss are critical ingredients to obtain powerful multi-sensory representations from models optimized to discern temporal synchronization of audio-video pairs. Without further finetuning, the resulting audio features achieve performance superior or comparable to the state-of-the-art on established audio classification benchmarks (DCASE2014 and ESC-50). At the same time, our visual subnet provides a very effective initialization to improve the accuracy of video-based action recognition models: compared to learning from scratch, our self-supervised pretraining yields a remarkable gain of +16.7% in action recognition accuracy on UCF101 and a boost of +13.0% on HMDB51.

Motivation & Objective

• To learn general-purpose audio and video representations using the natural correlation between audio and video signals.
• To address the challenge of learning robust multi-sensory features without human-annotated data.
• To improve performance on downstream audio and video recognition tasks through self-supervised pretraining.
• To investigate whether temporal alignment of audio and video can serve as a strong supervisory signal for representation learning.

Proposed method

• The method uses a contrastive loss to train a model to distinguish between positive (synchronized) and negative (asynchronized) audio-video pairs.
• A calibrated curriculum learning scheme is applied, gradually increasing the difficulty of negative examples during training.
• Negative samples are carefully selected to ensure they are semantically relevant but temporally misaligned, enhancing feature discrimination.
• The model jointly optimizes audio and video encoders, sharing a shared projection head for contrastive learning.
• Temporal synchronization is used as a self-supervisory signal, eliminating the need for manual annotations.
• The approach enables end-to-end training of audio and video encoders using only raw video and audio inputs.

Experimental results

Research questions

• RQ1 Can temporal synchronization between audio and video serve as an effective self-supervisory signal for learning general representations?
• RQ2 How does curriculum learning impact the quality of self-supervised audio-visual representations?
• RQ3 What is the effect of negative sampling strategy on contrastive learning performance in audio-visual representation learning?
• RQ4 To what extent can self-supervised pretraining improve downstream action recognition and audio classification accuracy?
• RQ5 How do the learned audio and video features compare to state-of-the-art methods on standard benchmarks?

Key findings

• The self-supervised audio features achieve performance superior or comparable to state-of-the-art on DCASE2014 and ESC-50 audio classification benchmarks.
• The visual features learned via self-supervised pretraining improve action recognition accuracy by +16.7% on UCF101 compared to training from scratch.
• The method yields a +13.0% accuracy gain on HMDB51 when using the pre-trained visual features for fine-tuning.
• The combination of contrastive loss, curriculum learning, and careful negative sampling is critical for achieving strong performance.
• The approach generalizes well, enabling strong zero-shot transfer performance without any task-specific fine-tuning.
• The results demonstrate that temporal alignment is a powerful signal for multi-sensory representation learning in videos.