[论文解读] Learning Features of Music from Scratch

John Thickstun, Zaïd Harchaoui+1 more|arXiv (Cornell University)|Nov 29, 2016

Music and Audio Processing参考文献 26被引用 115

一句话总结

本文介绍 MusicNet，一个用于学习音乐表征的大型标注数据集，并对端到端和基于声谱图的模型进行多标签音符预测的基准测试，结果表明学习的特征可以与声谱图相当，且 CNN/MLP 方法在 MusicNet 上表现出强劲性能。

ABSTRACT

MusicNet is a collection of 330 freely-licensed classical music recordings, together with over 1 million annotated labels indicating the precise time of each note in every recording, the instrument that plays each note, and the note's position in the metrical structure of the composition. The labels are acquired from musical scores aligned to recordings by dynamic time warping. The labels are verified by trained musicians; we estimate a labeling error rate of 4%. We offer the MusicNet labels to the machine learning and music communities as a resource for training models and a common benchmark for comparing results. This dataset was introduced in the paper "Learning Features of Music from Scratch." [1] This repository consists of 3 top-level files: <strong>musicnet.tar.gz</strong> - This file contains the MusicNet dataset itself, consisting of PCM-encoded audio wave files (.wav) and corresponding CSV-encoded note label files (.csv). The data is organized according to the train/test split described and used in "Invariances and Data Augmentation for Supervised Music Transcription". [2] <strong>musicnet_metadata.csv</strong> - This file contains track-level information about recordings contained in MusicNet. The data and label files are named with MusicNet ids, which you can use to cross-index the data and labels with this metadata file. <strong>musicnet_midis.tar.gz</strong> - This file contains the reference MIDI files used to construct the MusicNet labels. A PyTorch interface for accessing the MusicNet dataset is available on GitHub. For an audio/visual introduction and summary of this dataset, see the MusicNet inspector, created by Jong Wook Kim. The audio recordings in MusicNet consist of Creative Commons licensed and Public Domain performances, sourced from the Isabella Stewart Gardner Museum, the European Archive Foundation, and Musopen. The provenance of specific recordings and midis are described in the metadata file. [1] Learning Features of Music from Scratch. John Thickstun, Zaid Harchaoui, and Sham M. Kakade. In International Conference on Learning Representations (ICLR), 2017. ArXiv Report. <pre><code>@inproceedings{thickstun2017learning, title={Learning Features of Music from Scratch}, author = {John Thickstun and Zaid Harchaoui and Sham M. Kakade}, year={2017}, booktitle = {International Conference on Learning Representations (ICLR)} }</code></pre> [2] Invariances and Data Augmentation for Supervised Music Transcription. John Thickstun, Zaid Harchaoui, Dean P. Foster, and Sham M. Kakade. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2018. ArXiv Report. <pre><code>@inproceedings{thickstun2018invariances, title={Invariances and Data Augmentation for Supervised Music Transcription}, author = {John Thickstun and Zaid Harchaoui and Dean P. Foster and Sham M. Kakade}, year={2018}, booktitle = {International Conference on Acoustics, Speech, and Signal Processing (ICASSP)} }</code></pre>

研究动机与目标

• 创建一个大型、公开可用的标注古典音乐数据集（MusicNet），具备音符级对齐，以在音乐信息检索中实现监督学习。
• 定义音频片段上的多标签音符预测任务并建立评估协议。
• 对原始音频进行端到端音符预测的多体系架构基准测试（声谱图基、MLP、CNN）。
• 分析端到端模型学习的低层特征及其与传统声谱特征的比较。
• 研究数据集规模和窗口化如何影响对频率选择特征的学习。

提出的方法

• 用 34 小时的录音和 1,299,329 个标注音符，跨越10位作曲家和11种乐器，构建 MusicNet。
• 定义一个多标签分类设置，让每个音频片段标注128个乐器/音符组合。
• 在原始音频和基于声谱图的特征上训练模型，包括端到端神经网络和卷积网络。
• 学习低层、频率选择性的滤波器，呈现类似声谱表示的特征。
• 使用在保留测试集上的精确度、召回率和平均精度进行评估。
• 探索窗口大小效应和频率截止对学习到的时频权衡的影响。

实验结果

研究问题

• RQ1 大型标注的古典音乐数据集是否能够为音符转录提供有效的有监督学习？
• RQ2 端到端模型是否学习到可与声谱图相比的频率选择性特征以用于音符预测？
• RQ3 不同架构（MLP、CNN）在 MusicNet 的多标签音符预测任务上表现如何？
• RQ4 窗口大小和低频聚焦对对齐与转录性能有何影响？
• RQ5 学习到的特征在多大程度上优于或接近传统声谱表示？

主要发现

• MusicNet 提供了 34 小时的标注古典音乐和超过 129.3 万个时序标签，为音符预测提供了有监督学习的基础。
• 端到端模型学习出类似声谱的频率选择滤波器，在某些设置下可略微超过声谱特征的性能。
• 带对数-ReLU 激活的两层网络恢复出可解释的、受调制的正弦权重，类似局部光谱模式。
• 卷积和基于 MLP 的模型实现了可观的性能，其中 CNN（步幅 64、样本数 16,384）在表3中达到最高的平均精度 67.8%。
• 学习到的特征与传统声谱在性能上具有竞争力，窗口大小和特征中的低频强调会影响性能。
• 研究强调大规模数据有利于恢复有用的音乐表征。