[Paper Review] MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation

Li-Chia Yang, Szu-Yu Chou+1 more|arXiv (Cornell University)|Mar 31, 2017

Music Technology and Sound Studies31 references296 citations

TL;DR

MidiNet introduces a CNN-GAN with a 2-D conditioner to generate bar-by-bar melodies in the symbolic domain, showing comparable realism to MelodyRNN and higher perceived creativity, especially with chord conditioning.

ABSTRACT

Most existing neural network models for music generation use recurrent neural networks. However, the recent WaveNet model proposed by DeepMind shows that convolutional neural networks (CNNs) can also generate realistic musical waveforms in the audio domain. Following this light, we investigate using CNNs for generating melody (a series of MIDI notes) one bar after another in the symbolic domain. In addition to the generator, we use a discriminator to learn the distributions of melodies, making it a generative adversarial network (GAN). Moreover, we propose a novel conditional mechanism to exploit available prior knowledge, so that the model can generate melodies either from scratch, by following a chord sequence, or by conditioning on the melody of previous bars (e.g. a priming melody), among other possibilities. The resulting model, named MidiNet, can be expanded to generate music with multiple MIDI channels (i.e. tracks). We conduct a user study to compare the melody of eight-bar long generated by MidiNet and by Google's MelodyRNN models, each time using the same priming melody. Result shows that MidiNet performs comparably with MelodyRNN models in being realistic and pleasant to listen to, yet MidiNet's melodies are reported to be much more interesting.

Motivation & Objective

• Investigate whether CNNs can effectively generate symbolic-domain melodies bar-by-bar.
• Develop a GAN framework (generator, discriminator) with a conditioning mechanism to incorporate prior musical information.
• Enable conditioning on previous bars and chord progressions to guide generation.
• Demonstrate model flexibility to extend to multi-track MIDI and varying conditioning.
• Provide reproducible implementation and baseline comparison with MelodyRNN.

Proposed method

• Represent each bar as an h-by-w matrix capturing note presence across time steps.
• Use a generator CNN (G) fed by random noise z to produce a bar-like 2-D score via transposed convolutions.
• Train a discriminator CNN (D) to distinguish real vs generated bar data using cross-entropy loss.
• Introduce a conditioner CNN that processes conditioning matrices (e.g., previous bars) and injects them into intermediate layers of G.
• Apply feature matching and one-sided label smoothing to stabilize GAN training.
• Compare three MidiNet variants: (1) melody-only with previous-bar conditioning, (2) melody with chord conditioning for stability, (3) melody with chord and previous-bar conditioning for creativity.

Experimental results

Research questions

• RQ1 Can a CNN-GAN generate realistic and pleasant melodies in the symbolic MIDI domain?
• RQ2 Does conditioning on previous bars improve temporal coherence and musical structure?
• RQ3 Does conditioning on chord progressions improve harmony-consistent melody generation?
• RQ4 How does MidiNet compare to established RNN-based baselines (MelodyRNN) in realism, pleasantness, and interest?
• RQ5 Is the approach extensible to multi-track music generation?

Key findings

• MidiNet Model 1 (previous-bar conditioning) yields melodies comparable to MelodyRNN in pleasantness and realism, with higher perceived interestingness by listeners.
• MidiNet Model 2 (adding chord conditioning) achieves the highest pleasantness and realism among studied variants for both listeners with and without musical background.
• Model 3 (strong 2-D conditioning with chords and previous-bars) produces more creative results by enforcing stronger cross-bar connections.
• Users considered MelodyRNN models generally less interesting due to repetitiveness or safety, while MidiNet variants provided more varied and potentially creative outputs.
• The study demonstrates that 2-D conditioning via a conditioner CNN effectively leverages temporal structure without recurrent architectures.