[Paper Review] Contrastive Unsupervised Learning for Audio Fingerprinting

Zhesong Yu, Xingjian Du+2 more|arXiv (Cornell University)|Oct 26, 2020

Music and Audio Processing19 references4 citations

TL;DR

This paper proposes a contrastive unsupervised learning framework for audio fingerprinting based on the MoCo (momentum contrast) method, which learns robust and discriminative audio embeddings by contrasting distorted versions of the same audio (positive pairs) against different audio tracks (negative pairs). The approach achieves state-of-the-art robustness to severe distortions like pitch shifting and speed change, with a hit rate of 84.54% on VoxCeleb2-Test and 100% generalization under unseen distortions.

ABSTRACT

The rise of video-sharing platforms has attracted more and more people to shoot videos and upload them to the Internet. These videos mostly contain a carefully-edited background audio track, where serious speech change, pitch shifting and various types of audio effects may involve, and existing audio identification systems may fail to recognize the audio. To solve this problem, in this paper, we introduce the idea of contrastive learning to the task of audio fingerprinting (AFP). Contrastive learning is an unsupervised approach to learn representations that can effectively group similar samples and discriminate dissimilar ones. In our work, we consider an audio track and its differently distorted versions as similar while considering different audio tracks as dissimilar. Based on the momentum contrast (MoCo) framework, we devise a contrastive learning method for AFP, which can generate fingerprints that are both discriminative and robust. A set of experiments showed that our AFP method is effective for audio identification, with robustness to serious audio distortions, including the challenging speed change and pitch shifting.

Motivation & Objective

• Address the challenge of audio identification in video-sharing platforms where audio tracks are heavily distorted via pitch shifting, speed change, and effects.
• Improve robustness of audio fingerprinting systems beyond existing methods like Shazam and SAMAF, especially under severe distortions.
• Leverage unsupervised contrastive learning to learn discriminative and robust audio representations without requiring labeled data.
• Enable generalization to unknown audio distortions by training with diverse distortions during pre-training.
• Develop a compact, scalable fingerprinting system suitable for industrial deployment with low storage overhead.

Proposed method

• Adapt the MoCo contrastive learning framework to audio fingerprinting by treating an audio track and its augmented, distorted versions as positive pairs.
• Treat different audio tracks as negative pairs to enforce representation discrimination during training.
• Use data augmentation via random audio degradation (e.g., pitch shift, speed change, compression, equalization) to generate positive samples.
• Train a shared encoder (e.g., VGG-16, ResNet-18) with a query and key branch, where keys are momentum-updated to stabilize training.
• Extract 256-dimensional embeddings via global average pooling and L2 normalization from Mel-spectrograms (128×200) of 2.5s audio segments.
• Use a contrastive loss to minimize distance between positive pairs and maximize distance between negative pairs during end-to-end training.

Experimental results

Research questions

• RQ1 Can contrastive unsupervised learning improve audio fingerprinting robustness under severe distortions such as pitch shifting and speed change?
• RQ2 How does the choice of backbone network (e.g., VGG vs. ResNet) affect fingerprinting performance in low-resource, unsupervised settings?
• RQ3 To what extent can a contrastive learning-based AFP system generalize to unseen audio distortions not seen during training?
• RQ4 How does the proposed method compare to existing systems like Shazam and SAMAF in terms of hit rate under realistic distortions?
• RQ5 What is the storage efficiency and scalability of the learned fingerprints for large-scale industrial deployment?

Key findings

• The VGG-16 model pre-trained on ImageNet achieved the highest hit rate of 84.54% on the VoxCeleb2-Test set, outperforming ResNet-based models.
• The system achieved 100% hit rate on both VoxCeleb2-Test and Music-Test under equalization and compression distortions not used during training, indicating strong generalization.
• The Shazam-based baseline performed poorly (0.00% hit rate) on the same test set, highlighting the limitations of traditional methods under heavy distortions.
• The model demonstrated strong performance on music data (99.80% hit rate with VGG-16 without ImageNet pretraining), suggesting high discriminability for music.
• Fingerprints are compact—approximately 400 KB per 3-minute track—making a 100M-track database feasible at ~400 GB storage, suitable for industrial use.
• Surprisingly, deeper networks like ResNet-50 did not outperform VGG-16, suggesting that feature semantics and model capacity do not always correlate with AFP performance.