[Paper Review] A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music

Adam P. Roberts, Jesse Engel+3 more|arXiv (Cornell University)|Mar 13, 2018

Music and Audio Processing42 references257 citations

TL;DR

The paper introduces MusicVAE, a hierarchical latent-variable model that uses a hierarchical decoder to effectively model long-term structure in music sequences, enabling better reconstruction, interpolation, and attribute manipulation than a flat decoder VAE.

ABSTRACT

The Variational Autoencoder (VAE) has proven to be an effective model for producing semantically meaningful latent representations for natural data. However, it has thus far seen limited application to sequential data, and, as we demonstrate, existing recurrent VAE models have difficulty modeling sequences with long-term structure. To address this issue, we propose the use of a hierarchical decoder, which first outputs embeddings for subsequences of the input and then uses these embeddings to generate each subsequence independently. This structure encourages the model to utilize its latent code, thereby avoiding the "posterior collapse" problem, which remains an issue for recurrent VAEs. We apply this architecture to modeling sequences of musical notes and find that it exhibits dramatically better sampling, interpolation, and reconstruction performance than a "flat" baseline model. An implementation of our "MusicVAE" is available online at http://g.co/magenta/musicvae-code.

Motivation & Objective

• Motivate why VAEs struggle with long sequential data and posterior collapse in recurrent VAEs.
• Propose a hierarchical decoder to encourage latent usage and capture long-range structure in music.
• Demonstrate improved reconstruction, interpolation, and attribute manipulation for music sequences over flat decoders.
• Show multi-stream (multi-instrument) modeling benefits in musical data.
• Provide quantitative and qualitative evaluations on large MIDI datasets to validate the approach.

Proposed method

• Use a bidirectional LSTM encoder to map an entire sequence to a single latent vector z.
• Introduce a hierarchical decoder where a conductor RNN first outputs per-subsequence embeddings, which initialize bottom-level decoder RNNs for each subsequence.
• Segment input sequences into U nonoverlapping subsequences and constrain the decoder so that long-range context must flow through conductor embeddings.
• Extend to multi-stream (trio) modeling with separate instrument decoders driven by the same conductor embeddings.
• Train with standard VAE objective augmented to reduce posterior collapse and leverage scheduled sampling for longer sequences.

Experimental results

Research questions

• RQ1 Can a hierarchical decoder prevent posterior collapse and improve modeling of long musical sequences compared to a flat decoder?
• RQ2 Does a hierarchical MusicVAE better reconstruct, interpolate, and generate musically coherent long sequences (16-bar and longer) than a flat baseline?
• RQ3 How does multi-stream modeling (melody, bass, drums) benefit learning structure in music sequences?
• RQ4 Are latent-space manipulations (interpolations and attribute vectors) meaningful and musically coherent for music data?

Key findings

• Hierarchical MusicVAE significantly improves reconstruction accuracy on longer sequences (16-bar melodies/drum patterns and multi-stream data) versus a flat decoder.
• Latent-space interpolations with the hierarchical model produce smoother, more coherent transitions between melodies than data-space interpolation or flat models.
• Attribute vector arithmetic in latent space yields predictable musical changes (e.g., density, syncopation) with controllable manipulations across examples.
• Listening studies show the hierarchical model’s samples are judged more musical than those from the flat baseline across melodies, trio, and drum tasks.
• The hierarchical model reduces the gap between teacher-forced and sampled reconstruction accuracy, indicating better utilization of the latent code and mitigated exposure bias.