[Paper Review] Evaluation of CNN-based Automatic Music Tagging Models

Minz Won, Andrés Ferraro+2 more|arXiv (Cornell University)|Jun 1, 2020

Music and Audio Processing16 references46 citations

TL;DR

This paper provides a consistent evaluation of state-of-the-art CNN-based music tagging models across three datasets (MTAT, MSD, MTG-Jamendo) and assesses robustness to input perturbations, with pre-trained PyTorch implementations and models released.

ABSTRACT

Recent advances in deep learning accelerated the development of content-based automatic music tagging systems. Music information retrieval (MIR) researchers proposed various architecture designs, mainly based on convolutional neural networks (CNNs), that achieve state-of-the-art results in this multi-label binary classification task. However, due to the differences in experimental setups followed by researchers, such as using different dataset splits and software versions for evaluation, it is difficult to compare the proposed architectures directly with each other. To facilitate further research, in this paper we conduct a consistent evaluation of different music tagging models on three datasets (MagnaTagATune, Million Song Dataset, and MTG-Jamendo) and provide reference results using common evaluation metrics (ROC-AUC and PR-AUC). Furthermore, all the models are evaluated with perturbed inputs to investigate the generalization capabilities concerning time stretch, pitch shift, dynamic range compression, and addition of white noise. For reproducibility, we provide the PyTorch implementations with the pre-trained models.

Motivation & Objective

• Motivate the need for comparable evaluations due to divergent experimental setups in prior work.
• Provide a unified evaluation of multiple CNN-based music tagging models under a common protocol.
• Assess model robustness to time-stretch, pitch-shift, dynamic range compression, and white noise.
• Offer open-source PyTorch implementations and pre-trained models to enable reproducibility.

Proposed method

• Review and implement a diverse set of CNN-based architectures for music tagging (FCN, MusicNN, Sample-level CNN, CRNN, Self-attention, Harmonic CNN, Short-chunk CNN, and variants).
• Standardize input representations (Mel-spectrograms with consistent FFT and overlap) and training setups across datasets.
• Evaluate models using macro ROC-AUC and PR-AUC, averaged over tags, with 16-chunk aggregation for chunk-based models.
• Investigate robustness by perturbing test audio with pitch shift, time stretch, dynamic range compression, and white noise using MUDA-based deformations.
• Provide pre-trained models and PyTorch implementations to facilitate reproducibility.

Experimental results

Research questions

• RQ1 Which CNN-based architectures provide the best performance for automatic music tagging on common benchmarks?
• RQ2 How do short-chunk training strategies compare to longer-input architectures in terms of accuracy and generalization?
• RQ3 What is the impact of model design choices (e.g., 3x3 filters, pooling, harmonic representations) on robustness to common audio perturbations?
• RQ4 Do domain-knowledge-inspired models (e.g., Musicnn, Harmonic CNN) maintain advantages as dataset size grows?
• RQ5 How do different evaluation protocols (song-level vs chunk-level training) affect reported performance across datasets?

Key findings

• Short-chunk CNN variants consistently achieve top performance across MTAT, MSD, and MTG-Jamendo.
• Harmonic CNN and Short-chunk CNN show strong generalization under input perturbations, outperforming others on most deformation types.
• Models trained on short audio excerpts (3.69s–5s) outperform those using longer segments (29.1s) in most datasets.
• Self-attention slightly outperforms CRNN among sequential models, but neither surpasses the best short-chunk CNN or Harmonic CNN.
• Musicnn is competitive on MTAT but can be outperformed on larger datasets by other architectures.
• CRC-based and music-domain knowledge-driven designs may lag when scaling to larger or more diverse datasets.