[Paper Review] Towards an efficient deep learning model for musical onset detection

Rong Gong, Xavier Serra|arXiv (Cornell University)|Jun 18, 2018

Music and Audio Processing17 references3 citations

TL;DR

This paper proposes a highly efficient 5-layer CNN for musical onset detection that achieves state-of-the-art performance with only 28.3% of the trainable parameters of the baseline model. The method is validated on two datasets—instrumental music and solo singing voice—demonstrating that transfer learning across diverse musical content fails without retraining, underscoring the need for open code and reproducible training pipelines.

ABSTRACT

In this paper, we propose an efficient and reproducible deep learning model for musical onset detection (MOD). We first review the state-of-the-art deep learning models for MOD, and identify their shortcomings and challenges: (i) the lack of hyper-parameter tuning details, (ii) the non-availability of code for training models on other datasets, and (iii) ignoring the network capability when comparing different architectures. Taking the above issues into account, we experiment with seven deep learning architectures. The most efficient one achieves equivalent performance to our implementation of the state-of-the-art architecture. However, it has only 28.3% of the total number of trainable parameters compared to the state-of-the-art. Our experiments are conducted using two different datasets: one mainly consists of instrumental music excerpts, and another developed by ourselves includes only solo singing voice excerpts. Further, inter-dataset transfer learning experiments are conducted. The results show that the model pre-trained on one dataset fails to detect onsets on another dataset, which denotes the importance of providing the implementation code to enable re-training the model for a different dataset. Datasets, code and a Jupyter notebook running on Google Colab are publicly available to make this research understandable and easy to reproduce.

Motivation & Objective

• To address reproducibility issues in deep learning-based musical onset detection by providing transparent hyperparameter tuning and full training code.
• To evaluate multiple deep learning architectures systematically for onset detection, focusing on efficiency and performance trade-offs.
• To investigate the limitations of transfer learning across different musical content types, such as instrumental vs. vocal performances.
• To compare onset selection methods—peak-picking and score-informed HMM—under varying model architectures.
• To promote open science by releasing datasets, code, and a runnable Jupyter notebook for immediate replication.

Proposed method

• The authors evaluate seven deep learning architectures, including CNNs, BiLSTMs, and temporal models, on two distinct datasets: Böck (instrumental) and a custom jingju (solo singing) dataset.
• A 5-layer CNN is selected as the most efficient architecture after comparing F1-scores and parameter counts, achieving comparable performance to the state-of-the-art with significantly fewer trainable parameters.
• Onset detection is performed using a standard three-step pipeline: audio representation (log-mel spectrograms), ODF computation via the deep model, and onset selection via peak-picking or score-informed HMM.
• Transfer learning is evaluated using four strategies: fine-tuning from pre-trained models, re-training, and feature extraction, all tested across the two datasets.
• The score-informed HMM onset selection method is applied when musical score information is available, improving F1-scores over peak-picking by leveraging prior knowledge.
• All experiments are implemented with publicly available code, pre-trained models, and a Google Colab notebook to ensure full reproducibility.

Experimental results

Research questions

• RQ1 Can a deep learning model achieve state-of-the-art onset detection performance with significantly fewer parameters than existing models?
• RQ2 Why does transfer learning fail when transferring between datasets with different musical content, such as instrumental music and solo singing?
• RQ3 How do different onset selection methods—peak-picking versus score-informed HMM—affect final detection performance?
• RQ4 To what extent do hyperparameter choices and model architecture impact onset detection performance when training from scratch?
• RQ5 How can reproducibility in musical onset detection research be improved through open data, code, and detailed implementation documentation?

Key findings

• The 5-layer CNN achieves an F1-score of 86.52% on the Böck dataset and 76.68% on the jingju dataset, matching the performance of the state-of-the-art model while using only 28.3% of its trainable parameters.
• Pre-trained models from one dataset (e.g., Böck) fail drastically when applied to another (e.g., jingju), achieving only 38.73% and 34.28% F1-scores respectively, indicating domain-specific onset patterns.
• The score-informed HMM method improves F1-scores by a significant margin over peak-picking, especially on the jingju dataset, where it reaches 83.01% compared to 76.68% with peak-picking.
• Transfer learning strategies such as fine-tuning and feature extraction do not improve performance on the target dataset, suggesting that onset patterns differ fundamentally between instrumental and vocal music.
• The lack of code availability and insufficient hyperparameter details in prior work hinders reproducibility, as shown by the failure of transfer learning and the need for retraining on new datasets.
• The authors confirm that model performance is highly sensitive to training data distribution, and retraining is essential when adapting models to new musical content.