[Paper Review] dMelodies: A Music Dataset for Disentanglement Learning

Ashis Pati, Siddharth Gururani+1 more|arXiv (Cornell University)|Jul 29, 2020

Music and Audio Processing42 references4 citations

TL;DR

This paper introduces dMelodies, a large-scale symbolic music dataset of 1.3 million 2-bar monophonic melodies with nine disentangled latent factors (ordinal, categorical, binary). It enables benchmarking of disentanglement algorithms in music, revealing that methods effective on image datasets like dSprites underperform on music data, highlighting domain-specific challenges and the need for music-specific disentanglement research.

ABSTRACT

Representation learning focused on disentangling the underlying factors of variation in given data has become an important area of research in machine learning. However, most of the studies in this area have relied on datasets from the computer vision domain and thus, have not been readily extended to music. In this paper, we present a new symbolic music dataset that will help researchers working on disentanglement problems demonstrate the efficacy of their algorithms on diverse domains. This will also provide a means for evaluating algorithms specifically designed for music. To this end, we create a dataset comprising of 2-bar monophonic melodies where each melody is the result of a unique combination of nine latent factors that span ordinal, categorical, and binary types. The dataset is large enough (approx. 1.3 million data points) to train and test deep networks for disentanglement learning. In addition, we present benchmarking experiments using popular unsupervised disentanglement algorithms on this dataset and compare the results with those obtained on an image-based dataset.

Motivation & Objective

• Address the lack of standardized, diverse datasets for disentanglement learning in music, as most existing work relies on image-based benchmarks.
• Provide a systematically constructed music dataset with well-defined, independent latent factors of variation to support reproducible research in music representation learning.
• Enable evaluation of disentanglement algorithms on music data to assess their generalizability beyond computer vision domains.
• Investigate the performance gap between image-based and music-based disentanglement methods using benchmarking on both dMelodies and dSprites.
• Facilitate future research on semi-supervised and supervised disentanglement methods tailored to music generation and analysis tasks.

Proposed method

• Generate a symbolic music dataset of 2-bar monophonic melodies using algorithmic rules to ensure independence across nine latent factors: pitch class, octave, duration, rhythm, articulation, dynamics, key, time signature, and arpeggiation direction.
• Design the dataset to include diverse factor types: ordinal (e.g., duration), categorical (e.g., key), and binary (e.g., arpeggiation direction), enabling comprehensive disentanglement evaluation.
• Construct the dataset with approximately 1.3 million unique data points, scalable for training deep neural networks used in disentanglement learning.
• Apply popular unsupervised disentanglement models—β-VAE, FactorVAE, and β-VAE with residual connections—on both dMelodies and dSprites for comparative benchmarking.
• Use metrics such as Mutual Information Gap (MIG) and reconstruction loss to evaluate disentanglement quality and model fidelity across both datasets.
• Conduct factor-wise analysis of disentanglement performance to identify which musical attributes (e.g., rhythm, pitch) are more easily disentangled than others (e.g., arpeggiation direction).

Experimental results

Research questions

• RQ1 To what extent do state-of-the-art unsupervised disentanglement algorithms trained on image data generalize to symbolic music data?
• RQ2 How does the performance of disentanglement models differ between image-based (dSprites) and music-based (dMelodies) datasets in terms of disentanglement and reconstruction?
• RQ3 Which musical factors of variation (e.g., pitch, rhythm, dynamics) are most amenable to disentanglement using current deep generative models?
• RQ4 How sensitive are disentanglement performance and reconstruction fidelity to hyperparameter tuning in the music domain compared to the image domain?
• RQ5 What role do factor type (categorical, ordinal, binary) and sparsity of effect (e.g., arpeggiation affecting only part of the melody) play in disentanglement performance?

Key findings

• Disentanglement methods that perform well on image-based datasets like dSprites show significantly reduced performance on the dMelodies music dataset, indicating poor cross-domain generalization.
• Hyperparameter tuning has a much stronger impact on reconstruction and disentanglement performance in music (dMelodies) than in images (dSprites), where reconstruction remains stable across hyperparameter settings.
• The β-VAE model achieves the highest reconstruction accuracy on dMelodies, with rhythm and octave factors showing the best disentanglement, while arpeggiation direction performs the worst, likely due to its binary nature.
• Factor-wise MIG analysis reveals that continuous and structured factors (e.g., pitch, duration) are more easily disentangled than binary or sparsely affecting factors (e.g., arpeggiation direction).
• The performance gap between dSprites and dMelodies is not solely due to data scale, as both datasets contain ~1.3 million samples, suggesting architectural or inductive bias mismatches between domains.
• The results underscore the need for domain-specific disentanglement methods in music, as current unsupervised approaches are not directly transferable from vision to music.