[Paper Review] Generative Modelling for Controllable Audio Synthesis of Expressive Piano Performance

Hao Tan, Yin-Jyun Luo+1 more|arXiv (Cornell University)|Jun 16, 2020

Music and Audio Processing11 references4 citations

TL;DR

This paper proposes a controllable neural audio synthesizer based on Gaussian Mixture Variational Autoencoders (GM-VAE) that generates expressive piano performances from onset rolls, enabling fine-grained style morphing over time by conditioning on latent variables for articulation and dynamics. The model achieves realistic audio synthesis with gradual transitions between staccato/legato and soft/loud styles, as well as performance style transfer by inferring style factors from reference audio.

ABSTRACT

We present a controllable neural audio synthesizer based on Gaussian Mixture Variational Autoencoders (GM-VAE), which can generate realistic piano performances in the audio domain that closely follows temporal conditions of two essential style features for piano performances: articulation and dynamics. We demonstrate how the model is able to apply fine-grained style morphing over the course of synthesizing the audio. This is based on conditions which are latent variables that can be sampled from the prior or inferred from other pieces. One of the envisioned use cases is to inspire creative and brand new interpretations for existing pieces of piano music.

Motivation & Objective

• To develop a neural audio synthesizer that generates realistic piano performances directly from MIDI onset rolls, bypassing the need for frame- and velocity-level MIDI data.
• To enable fine-grained, controllable manipulation of expressive performance features such as articulation and dynamics in the audio domain.
• To support creative applications like gradual style morphing and performance style transfer by disentangling expressive factors in latent space.
• To overcome limitations of traditional synthesis methods that rely on stitched note samples or fixed piano simulations.

Proposed method

• The model uses a conditional VAE framework with a Gaussian mixture prior over two latent spaces: one for articulation (staccato/legato) and one for dynamics (loud/soft), enabling disentangled control.
• Latent variables are conditioned on binary style sequences derived from audio data: onset duration for articulation and average velocity for dynamics.
• The generation network employs a two-layer bidirectional LSTM to synthesize Mel-spectrograms from onset rolls and style-conditioned latent vectors.
• The inference network estimates latent style representations from reference audio, enabling performance style transfer by transferring style from a reference piece.
• A joint loss function optimizes reconstruction, KL divergence with respect to mixture priors, and additional cross-entropy losses for style prediction.
• WaveGlow is used to invert the generated Mel-spectrograms into high-fidelity audio, ensuring fast and high-quality inference.

Experimental results

Research questions

• RQ1 Can a generative model achieve fine-grained, continuous control over expressive piano performance features like articulation and dynamics in the audio domain?
• RQ2 How can latent representations be structured to enable smooth style morphing between different performance styles over time?
• RQ3 To what extent can a model transfer the expressive style of one piece to another while preserving the original musical content?
• RQ4 Can the model generate realistic audio performances without requiring full velocity- or frame-level MIDI input?

Key findings

• The model successfully generates realistic piano performances from onset rolls alone, achieving high perceptual quality without requiring full MIDI velocity or frame information.
• Gradual style morphing between staccato and legato articulation is achieved by linear interpolation in the latent space of articulation, resulting in notes that sustain progressively longer over time.
• Similarly, morphing from soft to loud dynamics results in increasing amplitude and higher-frequency energy, perceived as greater auditory energy.
• Performance style transfer is effective: a Baroque-era piece can be rendered in the expressive style of a Romantic-era piece, with accurate transfer of note duration and dynamic variation.
• The model preserves the original musical content during style transfer, as confirmed by aligned piano rolls and Mel-spectrograms showing consistent note onsets and pitch content.
• The use of a Gaussian mixture prior enables disentangled, interpretable control over expressive features, supporting creative exploration of new performance interpretations.