[Paper Review] Melodic Phrase Segmentation By Deep Neural Networks

Yixing Guan, Jinyu Zhao+3 more|arXiv (Cornell University)|Nov 14, 2018

Music and Audio Processing19 references4 citations

TL;DR

This paper proposes a deep learning approach for symbolic music melodic phrase segmentation using tailored label engineering and neural architectures like CNN-CRF, Bi-LSTM-CRF, and CNN. It achieves state-of-the-art performance with CNN-CRF yielding the highest F1 score (85.22±0.96) and fastest training, outperforming baseline models by effectively handling sparse labeling through sequence-aware labeling techniques and CRF-based label dependencies.

ABSTRACT

Automated melodic phrase detection and segmentation is a classical task in content-based music information retrieval and also the key towards automated music structure analysis. However, traditional methods still cannot satisfy practical requirements. In this paper, we explore and adapt various neural network architectures to see if they can be generalized to work with the symbolic representation of music and produce satisfactory melodic phrase segmentation. The main issue of applying deep-learning methods to phrase detection is the sparse labeling problem of training sets. We proposed two tailored label engineering with corresponding training techniques for different neural networks in order to make decisions at a sequential level. Experiment results show that the CNN-CRF architecture performs the best, being able to offer finer segmentation and faster to train, while CNN, Bi-LSTM-CNN and Bi-LSTM-CRF are acceptable alternatives.

Motivation & Objective

• To address the challenge of sparse labeling in supervised melodic phrase segmentation using deep neural networks.
• To explore and adapt various neural network architectures—CNN, Bi-LSTM, CRF—for symbolic music representation.
• To develop and evaluate two novel label engineering techniques that enhance sequential decision-making in phrase boundary detection.
• To compare the performance of end-to-end deep learning models with CRF layer integration against standard baselines.
• To establish a robust, learnable framework for automated music structure analysis in symbolic music.

Proposed method

• Proposes two label engineering techniques: linear-ascend and exponential-decay labeling to mitigate sparse labeling in training data.
• Combines deep neural networks (CNN, Bi-LSTM, U-Net) with CRF layers to model label dependencies and improve sequence-level predictions.
• Uses a supervised learning framework to optimize the conditional probability P(Y|X) via maximum likelihood estimation.
• Applies data augmentation to improve generalization, especially for CRF-based models.
• Employs a 5-fold cross-validation strategy on a custom symbolic music dataset for evaluation.
• Integrates CRF as a structured prediction layer to explicitly model relationships between adjacent phrase boundaries.

Experimental results

Research questions

• RQ1 Can deep neural networks effectively learn melodic phrase boundaries from symbolic music representations despite sparse labeling?
• RQ2 How do different neural architectures (CNN, Bi-LSTM, CRF) compare in performance and training efficiency for phrase segmentation?
• RQ3 To what extent do tailored label engineering techniques improve model performance on sparse sequence labeling tasks?
• RQ4 Does the integration of CRF layers significantly enhance prediction consistency and accuracy compared to standalone deep networks?
• RQ5 How does data augmentation affect the training stability and performance of CRF-based models?

Key findings

• The CNN-CRF model achieves the highest F1 score of 85.22±0.96 with the fastest training time, making it the optimal choice for this task.
• Bi-LSTM-CRF and Bi-LSTM-CNN models show strong performance but require significantly longer training times.
• Label engineering with linear-ascend and exponential-decay labeling improves model robustness, especially for CRF variants, which fail to converge with binary labeling.
• Data augmentation is critical for CRF models, as performance drops substantially without it—especially when using binary labels.
• The CNN-CRF model produces finer segmentation than ground truth in some cases, indicating it captures subtle phrase boundaries beyond human-annotated labels.
• The use of CRF layers significantly improves performance by modeling label dependencies, with CRF variants outperforming standalone CNN and Bi-LSTM models.