[Paper Review] SoundNet: Learning Sound Representations from Unlabeled Video

Yusuf Aytar, Carl Vondrick+1 more|arXiv (Cornell University)|Oct 27, 2016

Music and Audio ProcessingComputer Science29 references233 citations

TL;DR

SoundNet trains a deep convolutional network on raw audio by transferring visual knowledge from pre-trained vision models using unlabeled video as a bridge, achieving state-of-the-art results on acoustic scene classification datasets. It demonstrates that deeper networks benefit from large-scale unlabeled video and that learned sound representations encode high-level semantics.

ABSTRACT

We learn rich natural sound representations by capitalizing on large amounts of unlabeled sound data collected in the wild. We leverage the natural synchronization between vision and sound to learn an acoustic representation using two-million unlabeled videos. Unlabeled video has the advantage that it can be economically acquired at massive scales, yet contains useful signals about natural sound. We propose a student-teacher training procedure which transfers discriminative visual knowledge from well established visual recognition models into the sound modality using unlabeled video as a bridge. Our sound representation yields significant performance improvements over the state-of-the-art results on standard benchmarks for acoustic scene/object classification. Visualizations suggest some high-level semantics automatically emerge in the sound network, even though it is trained without ground truth labels.

Motivation & Objective

• Leverage large-scale unlabeled video to learn semantic sound representations.
• Transfer discriminative visual knowledge into the audio domain via a student-teacher framework.
• Demonstrate the effectiveness of deep, fully convolutional networks trained on raw audio for acoustic scene/object classification.

Proposed method

• Use a student-teacher setup where vision networks supervise a SoundNet audio network through KL-divergence between gk(y_i) and fk(x_i; θ).
• Train a deep fully convolutional network on raw audio waveforms to handle variable-length inputs.
• Employ a two-stage training: visual transfer from scene/object networks and subsequent audio classification via internal SoundNet representations with linear SVMs.
• Process over 2 million unlabeled videos from Flickr, converting to 22 kHz mono audio, and train with Adam optimizer.
• Experiment with eight-layer and five-layer SoundNet architectures to study depth effects.
• Visualize learned filters and hidden units to interpret high-level semantic detectors emerging in SoundNet.

Experimental results

Research questions

• RQ1 Can large-scale unlabeled video, via vision-to-sound transfer, yield semantically rich audio representations?
• RQ2 Does transferring from both object and scene visual models improve sound understanding?
• RQ3 How does network depth affect performance when trained with unlabeled video supervision?

Key findings

• SoundNet achieves state-of-the-art accuracy on acoustic scene classification benchmarks (e.g., DCASE, ESC-50, ESC-10) using features learned from unlabeled video.
• Eight-layer SoundNet with visual transfer significantly outperforms a five-layer version and baseline methods, indicating depth benefits when supervised by unlabeled video.
• Using KL-divergence for supervision (instead of L2) and combining ImageNet and Places as teachers improves performance.
• SoundNet features provide competitive accuracy when compared to vision features alone, and jointly with vision features yield modest gains on a multi-modal task.
• Visualization shows learned filters span diverse frequencies, and hidden units capture high-level audio concepts such as bird chirps or crowds cheering.