[Paper Review] Learning Features of Music from Scratch

John Thickstun, Zaïd Harchaoui+1 more|arXiv (Cornell University)|Nov 29, 2016

Music and Audio Processing26 references115 citations

TL;DR

The paper introduces MusicNet, a large labeled dataset for learning music representations, and benchmarks end-to-end and spectrogram-based models for multi-label note prediction, showing learned features can rival spectrograms and that CNN/MLP approaches achieve strong performance on MusicNet.

ABSTRACT

MusicNet is a collection of 330 freely-licensed classical music recordings, together with over 1 million annotated labels indicating the precise time of each note in every recording, the instrument that plays each note, and the note's position in the metrical structure of the composition. The labels are acquired from musical scores aligned to recordings by dynamic time warping. The labels are verified by trained musicians; we estimate a labeling error rate of 4%. We offer the MusicNet labels to the machine learning and music communities as a resource for training models and a common benchmark for comparing results. This dataset was introduced in the paper "Learning Features of Music from Scratch." [1] This repository consists of 3 top-level files: <strong>musicnet.tar.gz</strong> - This file contains the MusicNet dataset itself, consisting of PCM-encoded audio wave files (.wav) and corresponding CSV-encoded note label files (.csv). The data is organized according to the train/test split described and used in "Invariances and Data Augmentation for Supervised Music Transcription". [2] <strong>musicnet_metadata.csv</strong> - This file contains track-level information about recordings contained in MusicNet. The data and label files are named with MusicNet ids, which you can use to cross-index the data and labels with this metadata file. <strong>musicnet_midis.tar.gz</strong> - This file contains the reference MIDI files used to construct the MusicNet labels. A PyTorch interface for accessing the MusicNet dataset is available on GitHub. For an audio/visual introduction and summary of this dataset, see the MusicNet inspector, created by Jong Wook Kim. The audio recordings in MusicNet consist of Creative Commons licensed and Public Domain performances, sourced from the Isabella Stewart Gardner Museum, the European Archive Foundation, and Musopen. The provenance of specific recordings and midis are described in the metadata file. [1] Learning Features of Music from Scratch. John Thickstun, Zaid Harchaoui, and Sham M. Kakade. In International Conference on Learning Representations (ICLR), 2017. ArXiv Report. <pre><code>@inproceedings{thickstun2017learning, title={Learning Features of Music from Scratch}, author = {John Thickstun and Zaid Harchaoui and Sham M. Kakade}, year={2017}, booktitle = {International Conference on Learning Representations (ICLR)} }</code></pre> [2] Invariances and Data Augmentation for Supervised Music Transcription. John Thickstun, Zaid Harchaoui, Dean P. Foster, and Sham M. Kakade. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2018. ArXiv Report. <pre><code>@inproceedings{thickstun2018invariances, title={Invariances and Data Augmentation for Supervised Music Transcription}, author = {John Thickstun and Zaid Harchaoui and Dean P. Foster and Sham M. Kakade}, year={2018}, booktitle = {International Conference on Acoustics, Speech, and Signal Processing (ICASSP)} }</code></pre>

Motivation & Objective

• Create a large, publicly available labeled dataset of classical music with note-level alignments (MusicNet) to enable supervised learning in music informatics.
• Define a multi-label note prediction task on audio segments and establish an evaluation protocol.
• Benchmark multiple architectures (spectrogram-based, MLP, CNN) for end-to-end note prediction from raw audio.
• Analyze what low-level features end-to-end models learn and how they compare to traditional spectrogram features.
• Investigate how dataset size and windowing affect learning of frequency-selective features.

Proposed method

• Construct MusicNet with 34 hours of recordings and 1,299,329 labeled notes across 10 composers and 11 instruments.
• Define a multi-label classification setup where each audio segment is labeled with 128 instrument/note combinations.
• Train models on raw audio and on spectrogram-based features, including an end-to-end neural network and a convolutional network.
• Learn low-level, frequency-selective filters that resemble spectrogram-like representations.
• Evaluate using precision, recall, and average precision across a held-out test set.
• Explore window size effects and frequency cutoffs to understand time-frequency tradeoffs in learned features.

Experimental results

Research questions

• RQ1 Can a large labeled classical music dataset enable effective supervised learning for note transcription?
• RQ2 Do end-to-end models learn frequency-selective features comparable to spectrograms for note prediction?
• RQ3 How do different architectures (MLP, CNN) perform on MusicNet’s multi-label note prediction task?
• RQ4 What is the impact of window size and low-frequency focus on alignment and transcription performance?
• RQ5 To what extent do learned features outperform or approximate traditional spectrogram representations?

Key findings

• MusicNet provides 34 hours of labeled classical music with over 1.29 million temporal labels, enabling supervised learning for note prediction.
• End-to-end models learn frequency-selective filters that resemble spectrogram-like representations and can modestly outperform spectrogram features under certain settings.
• Two-layer networks with log-ReLU activations recover interpretable, modulated sinusoidal weights that resemble local spectral patterns.
• Convolutional and MLP-based models achieve substantial performance, with the CNN (64 stride, 16,384 samples) reaching the highest reported average precision in Table 3 (67.8%).
• Learned features are competitive with traditional spectrograms, with performance influenced by window size and low-frequency emphasis in the features.
• The study emphasizes that large data scales are beneficial for recovering useful music representations.