[Paper Review] The Chamber Ensemble Generator: Limitless High-Quality MIR Data via Generative Modeling

Yusong Wu, Josh Gardner+4 more|arXiv (Cornell University)|Sep 28, 2022

Music and Audio Processing8 citations

TL;DR

The paper proposes the Chamber Ensemble Generator (CEG), a pipeline combining a note-level generative model (Coconet) and an audio synthesis model (MIDI-DDSP) to produce limitless, high-quality, fully-annotated chamber ensemble music. This system generates the CocoChorales dataset, which significantly improves state-of-the-art performance in music transcription and source separation, especially for low-resource instruments.

ABSTRACT

Data is the lifeblood of modern machine learning systems, including for those in Music Information Retrieval (MIR). However, MIR has long been mired by small datasets and unreliable labels. In this work, we propose to break this bottleneck using generative modeling. By pipelining a generative model of notes (Coconet trained on Bach Chorales) with a structured synthesis model of chamber ensembles (MIDI-DDSP trained on URMP), we demonstrate a system capable of producing unlimited amounts of realistic chorale music with rich annotations including mixes, stems, MIDI, note-level performance attributes (staccato, vibrato, etc.), and even fine-grained synthesis parameters (pitch, amplitude, etc.). We call this system the Chamber Ensemble Generator (CEG), and use it to generate a large dataset of chorales from four different chamber ensembles (CocoChorales). We demonstrate that data generated using our approach improves state-of-the-art models for music transcription and source separation, and we release both the system and the dataset as an open-source foundation for future work in the MIR community.

Motivation & Objective

• Address the critical bottleneck of small, low-quality datasets in Music Information Retrieval (MIR), which limits model scalability and performance.
• Overcome the limitations of data augmentation and unsupervised learning by generating synthetic data with rich, structured annotations.
• Develop a scalable, open-source foundation for future MIR research by creating a large, high-fidelity dataset of chamber ensemble performances.
• Enable training of state-of-the-art models for low-resource instruments (e.g., oboe, bassoon) previously underserved due to data scarcity.
• Demonstrate that synthetic data from a structured generative pipeline can outperform real-world datasets in downstream MIR tasks.

Proposed method

• Use Coconet, trained on Bach Chorales, to generate four-part musical notation with instrument labels and expressive performance attributes (e.g., staccato, vibrato).
• Integrate MIDI-DDSP, trained on the URMP dataset, to synthesize high-fidelity audio from the generated note sequences, producing instrument stems and full mixtures.
• Construct a hierarchical generative pipeline where intermediate representations (note data, expression data, synthesis parameters) are preserved and modifiable.
• Apply controlled manipulations at intermediate stages—such as altering orchestration, tempo, microtiming, and pitch correction—to increase diversity and realism.
• Generate 240,000 unique performances to form the CocoChorales dataset, including aligned audio mixtures, stems, MIDI files, and per-note performance annotations.
• Release both the CEG system and the CocoChorales dataset as open-source resources to accelerate future MIR research.

Experimental results

Research questions

• RQ1 Can a generative modeling pipeline produce synthetic MIR data with sufficient realism and rich annotations to outperform real-world datasets in downstream tasks?
• RQ2 To what extent can synthetic data from a structured generative hierarchy improve performance in music transcription and source separation?
• RQ3 Can the Chamber Ensemble Generator generate high-quality audio for low-resource instruments (e.g., oboe, bassoon) that are underrepresented in existing datasets?
• RQ4 How does the inclusion of fine-grained annotations—such as note-level expression and synthesis parameters—affect model performance in MIR tasks?
• RQ5 Can dataset amplification via generative modeling serve as a scalable, open-source foundation for future MIR research?

Key findings

• The CocoChorales dataset, generated via the Chamber Ensemble Generator, enables a state-of-the-art music transcription model to achieve the best published results on the URMP dataset.
• A transcription model trained on CocoChorales achieved a 1.24 dB improvement in F0 accuracy over a baseline model trained on real data alone.
• Source separation models trained on CocoChorales achieved mean SI-SDR scores of 18.71 dB (flute), 17.28 dB (oboe), 21.01 dB (clarinet), and 20.68 dB (bassoon) on the woodwind ensemble test set.
• The CEG-generated data enabled training of a separation model on 360 hours of oboe data—far exceeding the less than 12 minutes available in the original URMP dataset—thus avoiding overfitting.
• The performance gains from CocoChorales were specific to the URMP dataset and did not degrade performance on other benchmark datasets, indicating no negative transfer.
• The results demonstrate that synthetic data with rich, structured annotations can effectively amplify real-world datasets and enable new capabilities in low-resource MIR tasks.