[Paper Review] Lakh Pianoroll Dataset

Hao‐Wen Dong, Wen-Yi Hsiao+2 more|arXiv (Cornell University)|Sep 19, 2017

Music Technology and Sound StudiesComputer Science151 citations

TL;DR

MuseGAN proposes multi-track sequential GANs for symbolic music generation, introduces three generation models (jamming, composer, hybrid), temporal structure handling, and the Lakh Pianoroll Dataset (LPD) derived from Lakh MIDI Dataset, with track-conditional generation and dedicated evaluation metrics.

ABSTRACT

Generating music has a few notable differences from generating images and videos. First, music is an art of time, necessitating a temporal model. Second, music is usually composed of multiple instruments/tracks with their own temporal dynamics, but collectively they unfold over time interdependently. Lastly, musical notes are often grouped into chords, arpeggios or melodies in polyphonic music, and thereby introducing a chronological ordering of notes is not naturally suitable. In this paper, we propose three models for symbolic multi-track music generation under the framework of generative adversarial networks (GANs). The three models, which differ in the underlying assumptions and accordingly the network architectures, are referred to as the jamming model, the composer model and the hybrid model. We trained the proposed models on a dataset of over one hundred thousand bars of rock music and applied them to generate piano-rolls of five tracks: bass, drums, guitar, piano and strings. A few intra-track and inter-track objective metrics are also proposed to evaluate the generative results, in addition to a subjective user study. We show that our models can generate coherent music of four bars right from scratch (i.e. without human inputs). We also extend our models to human-AI cooperative music generation: given a specific track composed by human, we can generate four additional tracks to accompany it. All code, the dataset and the rendered audio samples are available at https://salu133445.github.io/musegan/ .

Motivation & Objective

• Motivate generating multi-track, polyphonic symbolic music with temporal and inter-track coherence.
• Avoid simplifying assumptions (single-track/monophonic) by generating five tracks (bass, drums, guitar, piano, strings) using piano-roll representations.
• Develop three GAN-based generation models to capture jamming, composing, and hybrid inter-track dynamics.
• Extend the framework to track-conditional generation for human-AI cooperative music creation.
• Provide a new dataset (LPD) with preprocessing, and propose intra-track and inter-track objective metrics for evaluation.

Proposed method

• Represent multi-track music as bar-based piano-roll tensors to leverage CNNs.
• Adopt Wasserstein GAN with gradient penalty (WGAN-GP) as the core generative model.
• Define three generation schemes: jamming (independent private generators), composer (single shared generator), and hybrid (private generators with inter-track inputs).
• Incorporate temporal structure via two approaches: generation-from-scratch with a temporal generator feeding a bar generator, and track-conditional generation using an encoder to map a given track to a latent space.
• Assemble MuseGAN by combining inter-track and intra-track random vectors with per-track bar generators to produce bar-by-bar multi-track piano-rolls.
• Extend to track-conditional generation with an encoder to extract inter-track features from user-provided tracks.
• Preprocess data by mapping Lakh MIDI files to five-track piano-rolls (bass, drums, guitar, piano, strings) and segmenting into four-bar phrases for training.

Experimental results

Research questions

• RQ1 How can GAN-based models generate coherent multi-track, polyphonic symbolic music without reducing the problem to simpler subproblems?
• RQ2 What are effective architectures for modeling inter-track coordination and intra-track dynamics in multi-track music generation?
• RQ3 Can temporal structure be effectively learned to produce musically meaningful phrases (e.g., four-bar segments) in generated music?
• RQ4 Does track-conditional generation enable useful human-AI cooperative music composition or accompaniment?

Key findings

• Three GAN-based generation models (jamming, composer, hybrid) produce varying inter-track harmonic relations, with composer and hybrid often yielding stronger cross-track coherence.
• The jamming model tends to excel on intra-track metrics (less fragmentation), while composer/hybrid models better capture inter-track harmony (lower tonal distance TD).
• Temporal modeling enables bar-by-bar generation to form longer phrases, and track-conditional generation supports accompanying human-composed tracks.
• Objective metrics (EB, UPC, QN, DP, TD) provide insight into data sparsity, pitch usage, note quality, drum patterns, and inter-track harmony, guiding model assessment.
• A dataset of 50,266 four-bar phrases (LPD) with five tracks is provided, derived from Lakh MIDI Dataset and filtered for Rock, 4/4 time, and phrase structure.