[Paper Review] Deep Music Analogy Via Latent Representation Disentanglement

Ruihan Yang, Dingsu Wang+4 more|arXiv (Cornell University)|Jun 9, 2019

Music and Audio Processing25 references32 citations

TL;DR

The paper proposes an explicitly-constrained conditional variational autoencoder (EC2-VAE) to disentangle pitch and rhythm in 8-beat melodies conditioned on chords, enabling analogies by transferring latent factors and evaluating with objective metrics and a subjective study.

ABSTRACT

Analogy-making is a key method for computer algorithms to generate both natural and creative music pieces. In general, an analogy is made by partially transferring the music abstractions, i.e., high-level representations and their relationships, from one piece to another; however, this procedure requires disentangling music representations, which usually takes little effort for musicians but is non-trivial for computers. Three sub-problems arise: extracting latent representations from the observation, disentangling the representations so that each part has a unique semantic interpretation, and mapping the latent representations back to actual music. In this paper, we contribute an explicitly-constrained variational autoencoder (EC$^2$-VAE) as a unified solution to all three sub-problems. We focus on disentangling the pitch and rhythm representations of 8-beat music clips conditioned on chords. In producing music analogies, this model helps us to realize the imaginary situation of "what if" a piece is composed using a different pitch contour, rhythm pattern, or chord progression by borrowing the representations from other pieces. Finally, we validate the proposed disentanglement method using objective measurements and evaluate the analogy examples by a subjective study.

Motivation & Objective

• Motivation to enable analogy-based music generation by transferring high-level abstractions rather than raw observations.
• Develop a disentangled latent space where pitch and rhythm have explicit semantic meanings.
• Ensure disentanglement does not significantly degrade reconstruction quality and enables analogy at inference without analogous training data.

Proposed method

• Use an explicitly-constrained conditional variational autoencoder (EC2-VAE) to disentangle latent z into z_p (pitch) and z_r (rhythm).
• Condition both encoder and decoder on chords; include a rhythm-focused intermediate decoder task to enforce z_r semantics.
• Split latent z into two parts and connect z_r to a rhythm decoder trained with cross-entropy to match rhythm features.
• Prove the ELBO objective with disentanglement remains at least as tight as a standard conditional VAE under certain assumptions.
• Represent melodies as 8-beat sequences with 130-dim pitch-onset space and 3-dim rhythm features; chords provided as chroma-based condition.

Experimental results

Research questions

• RQ1 How can we explicitly disentangle pitch and rhythm in music representations within a VAE framework?
• RQ2 Can a disentangled model enable meaningful analogies by transferring pitch, rhythm, or chord representations between pieces?
• RQ3 Does explicit disentanglement compromise reconstruction quality, and can it remain close to the original ELBO objective?
• RQ4 What objective and subjective evidence shows the model’s effectiveness for generation via analogy?

Key findings

• EC2-VAE achieves explicit disentanglement by structuring the decoder to separate pitch and rhythm latent factors.
• Disentanglement preserves reconstruction quality and can be integrated with chord conditioning, enabling meaningful analogies.
• Objective metrics indicate effective separation of pitch and rhythm (Δz and F-score based augmentation query demonstrate high alignment with intended factors).
• Qualitative examples show successful analogy by replacing z_p (pitch) or z_r (rhythm) while preserving the other aspects and chord conditioning.
• Subjective evaluation shows the EC2-VAE variations are more creative and musical than a rule-based baseline and approach, but still lag behind original human-composed pieces in naturalness and overall musicality.