[Paper Review] End-to-end learning for music audio tagging at scale

Jordi Pons, Oriol Nieto+4 more|arXiv (Cornell University)|Nov 7, 2017

Music and Audio Processing19 references82 citations

TL;DR

The paper compares end-to-end waveform-based models with domain-knowledge spectrogram-based models for music auto-tagging across datasets of increasing size, showing waveform models outperform spectrogram models when trained on large-scale data (1M songs).

ABSTRACT

The lack of data tends to limit the outcomes of deep learning research, particularly when dealing with end-to-end learning stacks processing raw data such as waveforms. In this study, 1.2M tracks annotated with musical labels are available to train our end-to-end models. This large amount of data allows us to unrestrictedly explore two different design paradigms for music auto-tagging: assumption-free models - using waveforms as input with very small convolutional filters; and models that rely on domain knowledge - log-mel spectrograms with a convolutional neural network designed to learn timbral and temporal features. Our work focuses on studying how these two types of deep architectures perform when datasets of variable size are available for training: the MagnaTagATune (25k songs), the Million Song Dataset (240k songs), and a private dataset of 1.2M songs. Our experiments suggest that music domain assumptions are relevant when not enough training data are available, thus showing how waveform-based models outperform spectrogram-based ones in large-scale data scenarios.

Motivation & Objective

• Motivate automatic tagging of large music collections by learning from raw audio or spectrogram representations.
• Compare two design paradigms: waveform-based, assumption-free models vs. log-mal spectrogram-based, domain-knowledge-driven models.
• Evaluate performance as training data scales across multiple public/private datasets.
• Assess whether domain-knowledge constraints limit learning under large-data regimes.

Proposed method

• Define two front-ends: waveform (sample-level, small 1D filters) and spectrogram (log-mel, many vertical/horizontal filters) front-ends.
• Use a shared back-end: three 1D-CNN layers with residual connections, temporal pooling, and a dense output; global pooling enables variable-length inputs.
• Train end-to-end on 15-second patches; predict by averaging patch predictions for whole songs.
• Evaluate using ROC-AUC and PR-AUC for tag classification and RMSE for regression-like tags across datasets.
• Compare against a feature-extraction + gradient-boosted trees baseline to gauge end-to-end gains.

Experimental results

Research questions

• RQ1 How do waveform-based end-to-end models compare to spectrogram-based models for music auto-tagging across data regimes?
• RQ2 Does having large-scale training data diminish the need for domain-knowledge-based architectural constraints?
• RQ3 What is the impact of training data size on model performance for each front-end (waveform vs. spectrogram) across datasets?
• RQ4 Are waveform models able to match or surpass state-of-the-art spectrogram models when data is plentiful?

Key findings

• On 1.2M-songs, waveform models outperform spectrogram models in ROC-AUC, PR-AUC, and RMSE, and beat the baseline feature-based system.
• With 1M training songs, the waveform model achieves the highest ROC-AUC (92.50%) and PR-AUC (61.20%), outperforming the spectrogram model (ROC-AUC 92.17%, PR-AUC 59.92%).
• When trained on 100k or 500k songs, spectrogram models perform comparatively or slightly better, indicating domain-knowledge advantages with limited data.
• Across 1.2M, 1M, 500k, and 100k scales, waveform models show a stronger data-scaling benefit, suggesting less restrictive solution spaces are advantageous with more data.
• Spectrogram models align with state-of-the-art results on MSD when data is smaller, but waveform models reach superior performance with large data.