[Paper Review] PIANOTREE VAE: Structured Representation Learning for Polyphonic Music

Ziyu Wang, Yiyi Zhang+5 more|arXiv (Cornell University)|Aug 17, 2020

Music and Audio Processing39 references21 citations

TL;DR

PIANOTREE VAE proposes a hierarchical tree-structured variational autoencoder for polyphonic music representation learning, modeling music through a nested syntax of score, simu_note (simultaneous notes), and individual notes. It achieves semantically meaningful latent codes, improved reconstruction, and superior downstream music generation compared to baselines using MIDI events or standard VAEs.

ABSTRACT

The dominant approach for music representation learning involves the deep unsupervised model family variational autoencoder (VAE). However, most, if not all, viable attempts on this problem have largely been limited to monophonic music. Normally composed of richer modality and more complex musical structures, the polyphonic counterpart has yet to be addressed in the context of music representation learning. In this work, we propose the PianoTree VAE, a novel tree-structure extension upon VAE aiming to fit the polyphonic music learning. The experiments prove the validity of the PianoTree VAE via (i)-semantically meaningful latent code for polyphonic segments; (ii)-more satisfiable reconstruction aside of decent geometry learned in the latent space; (iii)-this model's benefits to the variety of the downstream music generation.

Motivation & Objective

• To address the lack of effective unsupervised representation learning for polyphonic music, which is more complex than monophonic music due to higher dimensionality and richer structure.
• To design a deep generative model that incorporates the hierarchical syntax of polyphonic music, such as the relationship between simultaneous notes and their temporal grouping.
• To improve representation learning by embedding inductive biases—specifically sparsity and hierarchical structure—into the VAE architecture.
• To demonstrate that structured latent representations lead to better downstream music generation and more interpretable interpolation in the latent space.

Proposed method

• The model uses a tree-structured architecture where each node represents a musical element: score, simu_note (simultaneous notes), or individual note, with hierarchical parent-child relationships.
• Each node is embedded via a recurrent neural network that encodes children into a parent or decodes a parent into its children, enabling bidirectional message passing.
• The model applies a variational autoencoder framework with a hierarchical prior and posterior, learning a structured latent space that captures musical semantics.
• It leverages sparsity in music input by modeling polyphonic music as sparse event sequences, reducing redundancy and improving training efficiency.
• The architecture supports both hierarchical encoding and decoding, enabling reconstruction and generation at multiple levels of musical abstraction.
• The latent code is derived from the mean of the posterior distribution over the hierarchical structure, used for downstream generation and interpolation.

Experimental results

Research questions

• RQ1 Can a hierarchical VAE architecture effectively learn meaningful and disentangled representations for polyphonic music, given its complex structure and high dimensionality?
• RQ2 Does incorporating inductive biases such as sparsity and hierarchical syntax improve the quality of music reconstruction and generation compared to standard VAEs or event-sequence models?
• RQ3 To what extent do the learned latent representations support smooth interpolation and semantically meaningful transitions in the latent space?
• RQ4 How does the structured representation from PianoTree VAE compare to MIDI-event-based or simu_note embedding-based representations in downstream music generation tasks?
• RQ5 Can the model generalize to longer-term music generation with improved coherence and creativity?

Key findings

• The PianoTree VAE produces semantically meaningful latent codes for polyphonic music segments, as validated by subjective evaluation of interpolation quality.
• The model achieves better reconstruction quality with more coherent geometry in the latent space compared to baseline models.
• Subjective evaluation shows that interpolations generated by PianoTree VAE are rated significantly higher in both musicality and transition smoothness than those from baselines.
• In downstream music generation, the use of latent vectors (z) and simu_note embeddings leads to significantly better ratings in creativity, naturalness, and musicality compared to MIDI-event tokens (p < 0.005).
• Long-form generation using the latent vector z produces more varied and coherent compositions than those using MIDI events or simu_note representations, which tend to repeat patterns.
• The model enables iterative generation of 16-measure compositions with improved structural coherence, especially when using the latent vector representation.