[Paper Review] Automatic tagging using deep convolutional neural networks

Keunwoo Choi, George Fazekas+1 more|arXiv (Cornell University)|Jun 1, 2016

Multimodal Machine Learning ApplicationsComputer Science18 references221 citations

TL;DR

This paper proposes fully convolutional networks (FCNs) with 2D convolutions to perform content-based automatic music tagging, showing mel-spectrogram inputs yield state-of-the-art results and deeper models benefit from more data.

ABSTRACT

We present a content-based automatic music tagging algorithm using fully convolutional neural networks (FCNs). We evaluate different architectures consisting of 2D convolutional layers and subsampling layers only. In the experiments, we measure the AUC-ROC scores of the architectures with different complexities and input types using the MagnaTagATune dataset, where a 4-layer architecture shows state-of-the-art performance with mel-spectrogram input. Furthermore, we evaluated the performances of the architectures with varying the number of layers on a larger dataset (Million Song Dataset), and found that deeper models outperformed the 4-layer architecture. The experiments show that mel-spectrogram is an effective time-frequency representation for automatic tagging and that more complex models benefit from more training data.

Motivation & Objective

• Demonstrate that fully convolutional neural networks can perform multi-label music tagging without fully connected layers.
• Evaluate input representations (mel-spectrogram, STFT, MFCC) for tagging performance.
• Assess how model depth (3–7 layers) impacts performance on different datasets.
• Show that mel-spectrograms outperform alternative representations for automatic tagging.
• Investigate how training data scale affects benefits of deeper architectures.

Proposed method

• Use fully convolutional networks composed of 3–7 convolutional layers with max-pooling to produce a 50-dimensional tag vector.
• Input representations include mel-spectrograms, STFT, and MFCCs; mel-spectrograms preferred for tagging.
• Train with sigmoid outputs and binary cross-entropy loss to handle multi-label data.
• Apply batch normalization and dropout to improve convergence and prevent overfitting.
• Evaluate architectures on MagnaTagATune (50 tags) and Million Song Dataset (top 50 tags) with AUC as the performance metric.
• Reference 2D convolutions to capture local time-frequency structure and temporal nonlinear aggregation over whole clips.

Experimental results

Research questions

• RQ1 How do FCN-based architectures perform on automatic music tagging across different input representations?
• RQ2 Does deeper network depth improve tagging performance, and does this depend on the size of the training data?
• RQ3 Is mel-spectrogram input superior to STFT or MFCC for automatic tagging with FCNs?
• RQ4 How does model depth interact with dataset scale (MagnaTagATune vs MSD) in multi-label tagging performance?

Key findings

• On MagnaTagATune, FCN-4 with mel-spectrogram input achieves AUC 0.894, outperforming several prior methods.
• Mel-spectrogram inputs consistently outperform STFT and MFCC inputs for the task on MagnaTagATune.
• Deeper architectures (FCN-5) show marginal gain over FCN-4 on MagnaTagATune, suggesting diminishing returns with limited data.
• On the Million Song Dataset, deeper models (FCN-5, FCN-6, FCN-7) significantly outperform FCN-4, indicating larger datasets benefit deeper networks.
• FCN-6 achieves the best MSD performance with AUC 0.851, while FCN-7 slightly underperforms FCN-6 in this setup.
• Overall, deeper models benefit from more training data, and mel-spectrograms are effective time-frequency representations for automatic tagging.