[Paper Review] Multi-Format Contrastive Learning of Audio Representations

Luyu Wang, Aäron van den Oord|arXiv (Cornell University)|Mar 11, 2021

Music and Audio Processing35 references30 citations

TL;DR

The paper shows that contrastive learning across audio formats (raw waveform and log-mel spectrogram) yields strong representations, achieving new state-of-the-art on AudioSet and ESC-50 without multi-modal data.

ABSTRACT

Recent advances suggest the advantage of multi-modal training in comparison with single-modal methods. In contrast to this view, in our work we find that similar gain can be obtained from training with different formats of a single modality. In particular, we investigate the use of the contrastive learning framework to learn audio representations by maximizing the agreement between the raw audio and its spectral representation. We find a significant gain using this multi-format strategy against the single-format counterparts. Moreover, on the downstream AudioSet and ESC-50 classification task, our audio-only approach achieves new state-of-the-art results with a mean average precision of 0.376 and an accuracy of 90.5%, respectively.

Motivation & Objective

• Motivate and evaluate whether multiple formats of a single audio modality can provide gains similar to multi-modal training.
• Develop a contrastive learning framework that maximizes agreement between raw audio and its spectral representation.
• Identify effective audio formats, augmentations, and architectures for self-supervised audio representation learning.
• Demonstrate downstream performance on AudioSet and ESC-50 to establish state-of-the-art results for unsupervised audio representation learning.

Proposed method

• Use a SimCLR-style contrastive objective to maximize agreement between two augmented views of the same audio in different formats (raw waveform vs spectral representations).
• Employ encoders suitable for each format (e.g., Conv-based or CNN/ResNet-style architectures) with a shared projection head.
• Apply lightweight augmentations for audio and spectrograms (audio mixing, time masking, frequency masking, frequency shift).
• Train with large batch sizes to provide many negatives for the contrastive loss and use a temperature parameter in the InfoNCE loss.
• Evaluate by freezing pretrained encoders and training shallow classifiers on downstream tasks (AudioSet mAP, ESC-50 accuracy).
• Experiment with single-format and dual-format setups to analyze benefits of cross-format agreement.

Experimental results

Research questions

• RQ1 Can learning representations by maximizing agreement between raw audio and spectral representations outperform single-format or single-modality baselines?
• RQ2 Which combinations of audio formats and augmentations yield the best downstream performance?
• RQ3 How do model architecture, batch size, temperature, and projection size affect contrastive learning for audio?
• RQ4 What are the downstream gains on AudioSet and ESC-50 when using dual-format self-supervised learning without labels?

Key findings

• Dual-format (raw waveform and log-mel spectrogram) training yields significant gains over single-format baselines.
• Raw audio and log-mel spectrogram pairing improves base model mAP by about 15% over raw-audio-only and by about 41% over log-mel-only on AudioSet.
• The best single-model results reach mAP 0.336 (waveform) and 0.329 (log-mel), with dual-format training achieving 0.368 (log-mel as eval) and 0.355 (waveform eval); concatenation of features from both formats reaches 0.376.
• ESC-50 downstream accuracy reaches 90.5% with dual-format training, setting a new state-of-the-art.
• Longer temporal crops (10s) perform worse due to multi-instance nature of AudioSet; 3–5s crops are preferred.
• Increasing batch size and projection latent size generally improves performance; larger models benefit more from dual-format setup.