[Paper Review] MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfer

Gino Brunner, Andres Konrad+2 more|arXiv (Cornell University)|Sep 20, 2018

Music and Audio Processing28 references63 citations

TL;DR

MIDI-VAE introduces a shared-latent-space variational autoencoder for polyphonic symbolic music that models dynamics and instrumentation and enables style transfer between genres (e.g., Classical to Jazz) as well as generation, interpolation, and mixtures.

ABSTRACT

We introduce MIDI-VAE, a neural network model based on Variational Autoencoders that is capable of handling polyphonic music with multiple instrument tracks, as well as modeling the dynamics of music by incorporating note durations and velocities. We show that MIDI-VAE can perform style transfer on symbolic music by automatically changing pitches, dynamics and instruments of a music piece from, e.g., a Classical to a Jazz style. We evaluate the efficacy of the style transfer by training separate style validation classifiers. Our model can also interpolate between short pieces of music, produce medleys and create mixtures of entire songs. The interpolations smoothly change pitches, dynamics and instrumentation to create a harmonic bridge between two music pieces. To the best of our knowledge, this work represents the first successful attempt at applying neural style transfer to complete musical compositions.

Motivation & Objective

• Motivate and develop a neural model to perform style transfer on symbolic music by exploiting dynamics (note velocities, durations) and instrumentation across multiple tracks.
• Learn a compact latent representation capturing joint distributions of pitch, velocity, and instrument assignments for multi-track music.
• Enable manipulation of existing pieces to change style (e.g., Classical to Jazz) while preserving melody, and facilitate generation, interpolation, and mixtures of pieces.

Proposed method

• Three parallel encoder–decoder pairs (pitch, velocity, instrument) share a common latent space to model the joint distribution of polyphonic music.
• A Variational Autoencoder with a β-weighted KL term enforces a prior on the latent z, promoting disentanglement.
• A style classifier is attached to the top latent dimensions to encode a latent style label, enabling style transfer by swapping style dimensions in z (z_style).
• Musical data are represented as piano rolls for pitch (60 pitches), velocity, and instrument rolls across tracks, with bars defined as one unit (16th-note granularity).
• The full loss combines pitch/pitch-velocity/instrument reconstruction terms, style prediction loss, and a KL divergence term (L_tot = λP H + λI H + λV MSE + λS H + (−β) DKL).
• Training uses GRU-based encoders/decoders and Adam optimization on a bar-by-bar symbolic representation extracted from MIDI files, with data split by genre pairs (e.g., Classic–Jazz).

Experimental results

Research questions

• RQ1 Can a shared latent space VAE capture the joint dynamics and instrumentation of multi-track symbolic music to allow style transfer between genres?
• RQ2 How effectively can a style classifier enforce a compact latent style representation and enable controllable style transfer across pieces?
• RQ3 What are the capabilities of MIDI-VAE for generation, interpolation, medleys, and mixtures of complete musical pieces while preserving musical coherence?
• RQ4 To what extent do pitch, velocity, and instrumentation contribute to successful style transfer between genres?

Key findings

• MIDI-VAE achieves style transfer by altering pitches, dynamics, and instrumentation to change a piece from one style to another, as validated by separate style classifiers.
• The model can interpolate between bars and generate medleys and mixtures of entire songs with smooth transitions in pitches, dynamics, and instrumentation.
• Latent space analyses (e.g., t-SNE) show clear separation of styles (e.g., Jazz vs Classical) and reveal dimensions correlated with style as well as other musical properties (note counts, pitch ranges).
• Pitch and velocity changes contribute to style transfer, while instrumentation changes show strong alignment with genre shifts; the melody is largely preserved.
• The approach represents an early, unaligned style-transfer application to complete musical compositions and is extendable to hierarchical models for longer-scale style features.