[Paper Review] Learning Interpretable Representation for Controllable Polyphonic Music Generation

Ziyu Wang, Dingsu Wang+2 more|arXiv (Cornell University)|Aug 17, 2020

Music and Audio Processing41 references20 citations

TL;DR

This paper proposes a VAE-based model that disentangles polyphonic music into interpretable chord and texture factors, enabling controllable generation through latent space manipulation. By incorporating rule-based chord recognition and chord-invariant convolutional networks, the method achieves high-quality, interpretable music generation with applications in style transfer, texture variation, and accompaniment arrangement, outperforming original human compositions in some style-transfer evaluations.

ABSTRACT

While deep generative models have become the leading methods for algorithmic composition, it remains a challenging problem to control the generation process because the latent variables of most deep-learning models lack good interpretability. Inspired by the content-style disentanglement idea, we design a novel architecture, under the VAE framework, that effectively learns two interpretable latent factors of polyphonic music: chord and texture. The current model focuses on learning 8-beat long piano composition segments. We show that such chord-texture disentanglement provides a controllable generation pathway leading to a wide spectrum of applications, including compositional style transfer, texture variation, and accompaniment arrangement. Both objective and subjective evaluations show that our method achieves a successful disentanglement and high quality controlled music generation.

Motivation & Objective

• To address the lack of interpretability in deep generative music models, where latent variables are difficult to control meaningfully.
• To enable precise control over compositional factors such as harmony and texture without relying on complex, task-specific architectures.
• To explore whether disentangled representations of chord (content) and texture (style) can serve as general-purpose controls for diverse music generation tasks.
• To evaluate the effectiveness of the disentangled representation through both objective metrics and subjective human evaluation.

Proposed method

• Employs a VAE framework with a dual-branch encoder that enforces disentanglement: one branch uses a rule-based chord recognizer to embed chord information into the first half of the latent space.
• The second branch treats music as 2D images and applies a chord-invariant convolutional network to extract texture features (rhythm, contour, voicing) into the second half of the latent space.
• Uses the PianoTree VAE decoder architecture to reconstruct polyphonic music hierarchically from the disentangled latent representation.
• Applies inductive biases through architectural design rather than loss functions, ensuring robust disentanglement without requiring additional training objectives.
• Enables control via latent space operations: swapping, sampling, and conditional prediction for downstream tasks.
• Leverages a hierarchical, hierarchical VAE-based decoder to reconstruct full 8-beat piano segments from the disentangled latent codes.

Experimental results

Research questions

• RQ1 Can chord and texture be effectively disentangled in the latent space of a VAE-based music generation model using architectural inductive biases?
• RQ2 To what extent do the disentangled latent factors enable controllable music generation across diverse tasks like style transfer and texture variation?
• RQ3 How do objective and subjective evaluations compare the quality of generated music, especially in style-transfer scenarios?
• RQ4 Can the model generate music that is rated higher than human-composed originals in specific control scenarios?

Key findings

• The model successfully disentangles chord and texture factors, as shown by objective metrics: chord representation is highly sensitive to pitch transposition, while texture representation responds strongly to rhythmic changes.
• In subjective evaluation, the best-performing style-transferred pieces were rated significantly higher than the original compositions (p < 0.005), indicating superior musicality and creativity in some cases.
• On average, transferred pieces were rated lower than originals, but the best transfers surpassed the original in both creativity and musicality.
• The model enables effective compositional style transfer by swapping latent codes, demonstrating that texture can be transferred independently of chord progression.
• Texture variation via sampling from the latent distribution successfully emulates 'Theme and Variations' form, preserving harmonic structure while altering rhythmic and melodic contours.
• Accompaniment arrangement via conditional prediction shows promise, suggesting the model can be extended to generate accompaniments from melodies using a downstream encoder-decoder setup.