[Paper Review] MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation using 1D and 2D Conditions.

Li-Chia Yang, Szu-Yu Chou+1 more|arXiv (Cornell University)|Mar 31, 2017

Music Technology and Sound Studies11 references32 citations

TL;DR

MidiNet is a 1D and 2D conditional convolutional GAN that generates symbolic music bar-by-bar, using harmonic context (1D chord condition) and sequential context (2D prior melody) to produce coherent, harmonic melodies. It outputs 16x128 matrices per bar (representing 16th-note resolution), enabling arbitrary-length, playable music sequences via concatenation.

ABSTRACT

In this paper, we present MidiNet, a deep convolutional neural network (CNN) based generative adversarial network (GAN) that is intended to provide a general, highly adaptive network structure for symbolic-domain music generation. The network takes random noise as input and generates a melody sequence one mea- sure (bar) after another. Moreover, it has a novel reflective CNN sub-model that allows us to guide the generation process by providing not only 1D but also 2D conditions. In our implementation, we used the intended chord of the current bar as a 1D condition to provide a harmonic context, and the melody generated for the preceding bar previously as a 2D condition to provide sequential information. The output of the network is a 16 by 128 matrix each time, representing the presence of each of the 128 MIDI notes in the generated melody sequence of that bar, with the smallest temporal unit being the sixteenth note. MidiNet can generate music of arbitrary number of bars, by concatenating these 16 by 128 matrices. The melody sequence can then be played back with a synthesizer. We provide example clips showing the effectiveness of MidiNet in generating harmonic music.

Motivation & Objective

• To develop a general, adaptive deep learning framework for symbolic-domain music generation.
• To enable conditional generation of music using both harmonic (1D) and sequential (2D) context.
• To generate coherent, harmonic melodies in real-time, bar-by-bar, with full control over structure and harmony.
• To support arbitrary-length music generation by concatenating generated bar outputs.

Proposed method

• The model uses a conditional GAN architecture with a generator that produces 16x128 matrices representing MIDI note events per bar at 16th-note resolution.
• A reflective CNN sub-model processes both 1D chord conditions (harmonic context) and 2D prior melody conditions (sequential context) to guide generation.
• The generator takes random noise and condition vectors as input, producing one bar at a time in a sequential manner.
• The discriminator evaluates the authenticity of generated bars, distinguishing real from generated sequences.
• The network is trained end-to-end to generate musically coherent, harmonic melodies with structural consistency.
• Generated bars are concatenated to form longer sequences, enabling arbitrary-length music output.

Experimental results

Research questions

• RQ1 Can a conditional GAN with 1D and 2D conditions generate musically coherent, harmonic melodies in a bar-by-bar fashion?
• RQ2 How effectively can 1D chord conditions and 2D prior melody conditions guide the generation of structurally consistent music?
• RQ3 Can the model produce long, continuous melodies by sequentially generating and concatenating bars?
• RQ4 To what extent does the reflective CNN sub-model improve generation quality compared to unconditioned or single-condition baselines?

Key findings

• MidiNet successfully generates harmonic, coherent music sequences by conditioning on both current chord progressions and previously generated melody.
• The use of 2D prior melody as a condition enables the model to maintain melodic continuity across bars.
• The model produces output in a standard symbolic format (16x128 matrices), directly playable via synthesizers.
• The bar-by-bar generation process allows for flexible, arbitrary-length music creation through concatenation.
• Example clips demonstrate the model's ability to generate musically plausible and harmonically consistent melodies.
• The reflective CNN sub-model effectively integrates multi-modal conditions (1D and 2D) to improve generation quality.