[Paper Review] Hierarchical Generative Modeling for Controllable Speech Synthesis

Wei-Ning Hsu, Yu Zhang+10 more|arXiv (Cornell University)|Oct 16, 2018

Speech Recognition and Synthesis26 references45 citations

TL;DR

The paper introduces GMVAE-Tacotron, a neural TTS model with two-level hierarchical latent variables (a discrete y_l and a continuous z_l) to controllably synthesize speech across speaker, style, and noise attributes using a Gaussian mixture prior within a VAE framework.

ABSTRACT

This paper proposes a neural sequence-to-sequence text-to-speech (TTS) model which can control latent attributes in the generated speech that are rarely annotated in the training data, such as speaking style, accent, background noise, and recording conditions. The model is formulated as a conditional generative model based on the variational autoencoder (VAE) framework, with two levels of hierarchical latent variables. The first level is a categorical variable, which represents attribute groups (e.g. clean/noisy) and provides interpretability. The second level, conditioned on the first, is a multivariate Gaussian variable, which characterizes specific attribute configurations (e.g. noise level, speaking rate) and enables disentangled fine-grained control over these attributes. This amounts to using a Gaussian mixture model (GMM) for the latent distribution. Extensive evaluation demonstrates its ability to control the aforementioned attributes. In particular, we train a high-quality controllable TTS model on real found data, which is capable of inferring speaker and style attributes from a noisy utterance and use it to synthesize clean speech with controllable speaking style.

Motivation & Objective

• Motivate controllable TTS for attributes rarely annotated in data (speaking style, accent, background noise, recording conditions).
• Propose a hierarchical variational autoencoder-based model to disentangle labeled speaker attributes from unlabeled latent attributes.
• Demonstrate learning of interpretable latent clusters and a sampling mechanism for diverse, controllable speech.
• Show effectiveness on real found data with varying recording conditions and unseen speakers.

Proposed method

• Extend Tacotron 2 to include two levels of latent variables modeled with Gaussian mixtures.
• Use p(y_l) as a uniform prior and p(z_l|y_l) as diagonal Gaussians to form a GMM prior over latent space.
• Infer posteriors q(z_l|X) and q(y_l|X) via encoder networks within a VAE framework and optimize ELBO.
• Introduce an observed attribute representation z_o to capture within-class variation tied to observed labels.
• Condition synthesizer on z_l, z_o, y_o (when available) to achieve disentangled control of latent attributes.
• Employ a Tacotron-2 based synthesizer with WaveRNN vocoder for efficient training and high-quality output.

Experimental results

Research questions

• RQ1 Can a two-level latent space (discrete y_l and continuous z_l) capture and disentangle latent speech attributes such as style, noise, and prosody?
• RQ2 Does a Gaussian mixture prior facilitate interpretable clustering of latent attributes and robust sampling for diverse speech synthesis?
• RQ3 Can the model learn to infer speaker-independent latent attributes from noisy or unseen speakers and still produce high-quality clean speech?
• RQ4 How does the GMVAE-Tacotron compare to GST, VAE, and baseline Tacotron-2 in naturalness and SNR across varied datasets?
• RQ5 Is it possible to sample and control speaking style and noise attributes independently while maintaining speaker identity?

Key findings

• GMVAE-Tacotron achieves interpretable latent clusters corresponding to attributes like clean/noisy conditions and speaker characteristics.
• The model demonstrates disentangled control over speaker, noise, and style attributes, enabling independent manipulation of each factor.
• On a noisy multi-speaker corpus, the GMVAE can synthesize clean speech for speakers with noisy training data, achieving higher MOS and SNR than baselines.
• The continuous latent space allows sampling with controllable attributes, yielding diverse yet natural prosody and speaking styles.
• Experiments show high consistency of mixture components with speaker groups (e.g., accents, gender) and per-dimension control over attributes like F0 and speed.