[Paper Review] Codified audio language modeling learns useful representations for music information retrieval

Rodrigo Castellon, Chris Donahue+1 more|arXiv (Cornell University)|Jul 12, 2021

Music and Audio Processing61 references22 citations

TL;DR

This paper demonstrates that codified audio language modeling (CALM), using representations from the Jukebox model pre-trained on 1 million songs, yields significantly stronger features for music information retrieval (MIR) than traditional tag-based pre-training. On average, CALM features improve performance by 30% across four MIR tasks—tagging, genre classification, key detection, and emotion recognition—particularly excelling in key detection, suggesting audio-based pre-training captures richer, more transferable representations than label-based methods.

ABSTRACT

We demonstrate that language models pre-trained on codified (discretely-encoded) music audio learn representations that are useful for downstream MIR tasks. Specifically, we explore representations from Jukebox (Dhariwal et al. 2020): a music generation system containing a language model trained on codified audio from 1M songs. To determine if Jukebox's representations contain useful information for MIR, we use them as input features to train shallow models on several MIR tasks. Relative to representations from conventional MIR models which are pre-trained on tagging, we find that using representations from Jukebox as input features yields 30% stronger performance on average across four MIR tasks: tagging, genre classification, emotion recognition, and key detection. For key detection, we observe that representations from Jukebox are considerably stronger than those from models pre-trained on tagging, suggesting that pre-training via codified audio language modeling may address blind spots in conventional approaches. We interpret the strength of Jukebox's representations as evidence that modeling audio instead of tags provides richer representations for MIR.

Motivation & Objective

• To investigate whether representations from a music generation model pre-trained via codified audio language modeling (CALM) are useful for discriminative MIR tasks.
• To compare the effectiveness of CALM-derived features against those from models pre-trained on manual tags, metadata, and contrastive learning.
• To determine if modeling raw audio directly, rather than relying on labels, leads to richer and more generalizable representations for MIR.
• To evaluate the potential of repurposing large-scale NLP-style models—originally designed for generation—for downstream discriminative MIR tasks.

Proposed method

• Pre-trained Jukebox model, which uses a Transformer language model on discrete audio codes from 1 million songs, is used to extract audio representations.
• Audio waveforms are first encoded into discrete tokens via a VQ-VAE, enabling application of NLP-style language modeling to audio.
• Shallow probing models (e.g., feedforward networks) are trained on downstream MIR tasks using the extracted features from Jukebox as input.
• The performance of these probes is compared to baselines using features from models pre-trained on tags, metadata, or via contrastive learning.
• Experiments are conducted on four MIR tasks: music tagging, genre classification, key detection, and emotion recognition.
• All experiments are reproducible using Docker containers and CodaLab worksheets, with code and model weights released for transparency.

Experimental results

Research questions

• RQ1 Can representations from a music generation model pre-trained via codified audio language modeling be effectively transferred to downstream MIR tasks?
• RQ2 How do CALM-derived features compare in performance to those from models pre-trained on manual tags, metadata, or contrastive learning?
• RQ3 Does pre-training on audio rather than labels lead to more robust and informative representations for MIR?
• RQ4 Is CALM particularly beneficial for tasks that are less correlated with tagging, such as key detection?

Key findings

• On average, representations from Jukebox’s CALM pre-training improve performance by 30% across four MIR tasks compared to tag-based pre-training.
• For key detection, CALM features significantly outperform tag-based features, indicating that audio modeling captures structural properties not well represented in tags.
• The CALM-based approach achieves competitive performance with state-of-the-art models on multiple MIR tasks, despite using only a single 12GB GPU for inference.
• The results suggest that modeling raw audio directly yields richer, more generalizable representations than label-based pre-training, especially for tasks not directly related to tagging.
• The success of CALM supports the broader hypothesis that large-scale audio modeling, inspired by NLP, can unlock new capabilities in MIR.
• The authors release code, Docker containers, and CodaLab worksheets to ensure full reproducibility and encourage further research in CALM-based MIR.