[Paper Review] MuseMorphose: Full-Song and Fine-Grained Piano Music Style Transfer with One Transformer VAE

Shih-Lun Wu, Yi‐Hsuan Yang|arXiv (Cornell University)|May 10, 2021

Music and Audio Processing4 citations

TL;DR

MuseMorphose proposes a unified Transformer VAE model for full-song and fine-grained piano music style transfer, using in-attention conditioning to enable bar-level control over rhythmic intensity and polyphony. It outperforms RNN-based baselines on key style transfer metrics by combining long-sequence generation with precise, disentangled attribute control.

ABSTRACT

Transformers and variational autoencoders (VAE) have been extensively employed for symbolic (e.g., MIDI) domain music generation. While the former boast an impressive capability in modeling long sequences, the latter allow users to willingly exert control over different parts (e.g., bars) of the music to be generated. In this paper, we are interested in bringing the two together to construct a single model that exhibits both strengths. The task is split into two steps. First, we equip Transformer decoders with the ability to accept segment-level, time-varying conditions during sequence generation. Subsequently, we combine the developed and tested in-attention decoder with a Transformer encoder, and train the resulting MuseMorphose model with the VAE objective to achieve style transfer of long pop piano pieces, in which users can specify musical attributes including rhythmic intensity and polyphony (i.e., harmonic fullness) they desire, down to the bar level. Experiments show that MuseMorphose outperforms recurrent neural network (RNN) based baselines on numerous widely-used metrics for style transfer tasks.

Motivation & Objective

• To bridge the strengths of Transformers (long-sequence modeling) and VAEs (controllable generation) for symbolic music generation.
• To enable fine-grained, segment-level (bar-level) control over musical attributes such as rhythmic intensity and polyphony during long music generation.
• To develop a single, unified model that supports both full-song generation and precise attribute transfer without auxiliary loss terms.
• To investigate the effectiveness of different conditioning mechanisms—pre-attention, in-attention, and post-attention—for controlling autoregressive Transformer decoders.
• To demonstrate that a VAE-trained Transformer model can achieve high-fidelity, diverse, and controllable music generation without relying on task-specific auxiliary objectives.

Proposed method

• Introduces in-attention conditioning, where time-varying conditioning vectors are injected into the attention mechanism of a Transformer decoder by modifying query, key, and value projections.
• Employs a Transformer encoder to map full music sequences into a latent space, and a conditional Transformer decoder to reconstruct the sequence using latent codes and segment-level conditions.
• Trains the model end-to-end with a VAE objective, incorporating cyclical KL annealing and free bits to stabilize training and improve disentanglement.
• Uses a vector-quantized latent space to support disentangled attribute control, enabling manipulation of specific musical features like polyphony and rhythmic intensity.
• Applies nucleus sampling with temperature and truncation adjustments to balance diversity and quality in generation.
• Evaluates conditioning mechanisms via ablation, showing in-attention outperforms pre- and post-attention in control and fluency.

Figure 1 : Visualizations of an original music excerpt (top) and one of its style-transferred generations by MuseMorphose (bottom). Digits in the upper and lower rows represent our target attributes being controlled, namely, bar-wise rhythmic intensity and polyphony classes, respectively.

Experimental results

Research questions

• RQ1 Can in-attention conditioning enable more effective and stable bar-level control over musical attributes in autoregressive Transformer decoders?
• RQ2 Does combining a Transformer encoder with a conditionally decodable Transformer decoder via VAE training yield superior performance in full-song style transfer compared to RNN-based VAEs?
• RQ3 To what extent can a single VAE-based model achieve high-fidelity, diverse, and independently controllable music generation without auxiliary loss terms?
• RQ4 How does the performance of MuseMorphose scale with increasing generation length, particularly in terms of attribute control and diversity?
• RQ5 Is the in-attention mechanism generalizable to other types of musical conditions, such as tonality or instrumentation, that vary less frequently than rhythmic intensity and polyphony?

Key findings

• The in-attention conditioning mechanism achieved the strongest performance in attribute control and sequence fluency, outperforming pre- and post-attention methods in ablation studies.
• MuseMorphose outperformed two RNN-based baselines on multiple standard style transfer metrics, including attribute control, fidelity, and diversity, without using auxiliary loss terms.
• The model maintained consistent performance across generation lengths from 8 to 96 bars, with only slight declines in attribute control and grooving diversity at longer lengths.
• The use of cyclical KL annealing and free bits in the VAE objective was critical to model success, enabling stable training and disentangled latent representations.
• The model demonstrated strong disentanglement of rhythmic intensity and polyphony, allowing independent manipulation of these attributes at the bar level.
• The framework is generalizable to NLP, with potential applications in sentiment control and text difficulty adjustment for educational content.

(a) Pre-attention conditioning