[Paper Review] Sample-level Deep Convolutional Neural Networks for Music Auto-tagging Using Raw Waveforms

Jongpil Lee, Ji Young Park+2 more|arXiv (Cornell University)|Mar 6, 2017

Music and Audio ProcessingComputer Science17 references104 citations

TL;DR

The paper proposes sample-level DCNNs that operate on raw waveforms for music auto-tagging, achieving state-of-the-art or comparable results to mel-spectrogram based methods on MTAT and MSD, and explores deep architectures (10+ layers) with small first-layer filters.

ABSTRACT

Recently, the end-to-end approach that learns hierarchical representations from raw data using deep convolutional neural networks has been successfully explored in the image, text and speech domains. This approach was applied to musical signals as well but has been not fully explored yet. To this end, we propose sample-level deep convolutional neural networks which learn representations from very small grains of waveforms (e.g. 2 or 3 samples) beyond typical frame-level input representations. Our experiments show how deep architectures with sample-level filters improve the accuracy in music auto-tagging and they provide results comparable to previous state-of-the-art performances for the Magnatagatune dataset and Million Song Dataset. In addition, we visualize filters learned in a sample-level DCNN in each layer to identify hierarchically learned features and show that they are sensitive to log-scaled frequency along layer, such as mel-frequency spectrogram that is widely used in music classification systems.

Motivation & Objective

• Motivate end-to-end learning directly from raw waveforms for music auto-tagging to address issues of log-amplitude compression and phase invariance.
• Introduce and evaluate sample-level DCNNs that use very small first-layer filters and increased depth to learn hierarchical audio representations.
• Compare sample-level raw waveform DCNNs against frame-level mel-spectrogram and frame-level raw waveform baselines and prior state-of-the-art results.
• Demonstrate that deeper architectures (beyond 10 layers) learned from raw waveforms can achieve competitive performance and visualize learned filters across layers.

Proposed method

• Introduce three CNN model configurations: frame-level mel-spectrogram, frame-level raw waveform, and sample-level raw waveform DCNNs.
• Replace bottom-layer processing with multiple small-filter, max-pooling modules to enable sample-level granularity (as small as 2-3 samples) and deeper networks.
• Systematically vary first-layer filter length and stride (2-3 samples) and depth (m^n with m in {2,3,4,5} and n up to 9–13) to study impact on AUC.
• Train with sigmoid outputs and binary cross-entropy loss, using batch normalization and ReLU, with dropout on the last conv layer, SGD with Nesterov momentum, and learning-rate scheduling.
• Evaluate on MagnaTagATune (MTAT) and Million Song Dataset (MSD), using 29.1-second trimmed songs at 22.05 kHz, with AUC as the primary metric.
• Visualize learned filters via gradient-ascent-based activation maximization to illustrate hierarchical spectral characteristics across layers.

Experimental results

Research questions

• RQ1 Can end-to-end sample-level DCNNs trained on raw waveforms achieve competitive auto-tagging performance compared to mel-spectrogram based approaches?
• RQ2 Does increasing depth with small first-layer filters improve performance for music auto-tagging on MTAT and MSD?
• RQ3 What are the characteristics of learned filters across layers when trained on raw waveforms, and how do they relate to frequency representation?
• RQ4 How do input segment length and first-layer stride affect performance in sample-level DCNNs?

Key findings

• Sample-level DCNNs with very small first-layer filters (as small as 2-3 samples) and greater depth yield AUC up to 0.9055 on MTAT with 59049-sample inputs, comparable to state-of-the-art mel-spectrogram results.
• For MTAT, the best m^n configuration used m=3 and n=9, achieving strong performance across input sizes and demonstrating depth benefits.
• On MSD, increasing the number of filters in the first convolution layer improves performance, with the model achieving 0.8812 AUC.
• Compared to frame-level mel-spectrogram models, sample-level raw waveform DCNNs achieve comparable results, and frame-level raw waveform models underperform without sufficient depth and expressive capacity.
• Visualization of learned filters shows progressively higher-centered frequency bands across layers, indicating hierarchical, frequency-aware representations similar to mel-scale tendencies.
• The sample-level approach demonstrates that deeper networks with small temporal filters can effectively learn polyphonic music representations from raw waveforms.