[Paper Review] Lead Sheet Generation and Arrangement by Conditional Generative Adversarial Network

Hao-Min Liu, Yi‐Hsuan Yang|arXiv (Cornell University)|Jul 30, 2018

Music and Audio Processing21 references4 citations

TL;DR

This paper proposes a conditional generative adversarial network (cGAN) for lead sheet arrangement, generating musically coherent multi-instrument arrangements from lead sheets using unpaired datasets. By introducing three symbolic-domain harmonic features—chroma piano-roll, chroma beats, and chord piano-roll—the model achieves superior performance, with chord piano-roll yielding the best perceptual results in user studies (MOS 3.5675 for overall feeling).

ABSTRACT

Research on automatic music generation has seen great progress due to the development of deep neural networks. However, the generation of multi-instrument music of arbitrary genres still remains a challenge. Existing research either works on lead sheets or multi-track piano-rolls found in MIDIs, but both musical notations have their limits. In this work, we propose a new task called lead sheet arrangement to avoid such limits. A new recurrent convolutional generative model for the task is proposed, along with three new symbolic-domain harmonic features to facilitate learning from unpaired lead sheets and MIDIs. Our model can generate lead sheets and their arrangements of eight-bar long. Audio samples of the generated result can be found at https://drive.google.com/open?id=1c0FfODTpudmLvuKBbc23VBCgQizY6-Rk

Motivation & Objective

• Address the gap in automatic music generation by introducing lead sheet arrangement as a new task between lead sheets and full MIDI.
• Overcome the lack of paired lead sheet and MIDI datasets by enabling conditional generation from unpaired data.
• Develop a recurrent convolutional cGAN architecture that generates both lead sheets and their multi-instrument arrangements.
• Design and evaluate three symbolic-domain harmonic features to bridge lead sheets and MIDIs for conditional generation.
• Improve temporal modeling in lead sheet generation by replacing convolutional layers with recurrent layers in the temporal generator.

Proposed method

• Propose a three-stage cGAN model: lead sheet generation, feature extraction, and arrangement generation, with conditional supervision.
• Use conditional GANs to generate five-track piano-rolls (melody, piano, guitar, strings, bass) conditioned on lead sheets.
• Introduce three harmonic features—chroma piano-roll, chroma beats, and chord piano-roll—computed from both lead sheets and MIDIs for cross-modal conditioning.
• Replace standard convolutional layers in temporal generators with recurrent layers to better model repetitive patterns in lead sheets.
• Adapt the convolutional GAN from Dong et al. for multi-track piano-roll generation, applying it to both lead sheet and arrangement generation.
• Train the model end-to-end using adversarial loss and perceptual metrics, with evaluation via objective metrics and user studies.

Experimental results

Research questions

• RQ1 Can a conditional GAN effectively generate musically coherent multi-instrument arrangements from unpaired lead sheets and MIDI datasets?
• RQ2 Which symbolic-domain harmonic representation—chroma piano-roll, chroma beats, or chord piano-roll—best preserves harmonic and rhythmic structure in lead sheet arrangement?
• RQ3 How does the use of recurrent layers in the temporal generator improve lead sheet generation compared to purely convolutional designs?
• RQ4 To what extent do the generated arrangements match human perception in terms of harmonicity, rhythmicity, and overall musicality?
• RQ5 Can lead sheet arrangement serve as a meaningful intermediate task between monophonic melody generation and full multi-track MIDI synthesis?

Key findings

• The model successfully generates eight-bar lead sheets and their arrangements with no empty bars, achieving stable training and high-quality output.
• The chord piano-roll feature outperforms chroma piano-roll and chroma beats in all perceptual metrics, with a mean opinion score (MOS) of 3.5675 for overall feeling in user studies.
• Chroma beats representation underperforms due to loss of temporal resolution, particularly in rhythmicity, despite good performance in harmonicity.
• The model achieves a tonal distance (TD) of 6.69 and a qualified note ratio (QN) of 0.84 at 5,000 iterations, indicating strong alignment with real lead sheet statistics.
• The use of recurrent layers in the temporal generator improves modeling of repetitive patterns in lead sheets, enhancing generation quality.
• User study results show that harmonicity has a stronger impact on overall musical feeling than rhythmicity, as evidenced by the significant MOS difference between chord-roll and other features.