[Paper Review] Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis

Rafael Valle, Kevin J. Shih+2 more|arXiv (Cornell University)|May 12, 2020

Speech Recognition and SynthesisComputer Science26 references81 citations

TL;DR

Flowtron presents an autoregressive flow-based TTS model with controllable speech variation and style transfer, achieving MOS comparable to state-of-the-art models and enabling latent-space manipulation for expressive speech.

ABSTRACT

In this paper we propose Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis with control over speech variation and style transfer. Flowtron borrows insights from IAF and revamps Tacotron in order to provide high-quality and expressive mel-spectrogram synthesis. Flowtron is optimized by maximizing the likelihood of the training data, which makes training simple and stable. Flowtron learns an invertible mapping of data to a latent space that can be manipulated to control many aspects of speech synthesis (pitch, tone, speech rate, cadence, accent). Our mean opinion scores (MOS) show that Flowtron matches state-of-the-art TTS models in terms of speech quality. In addition, we provide results on control of speech variation, interpolation between samples and style transfer between speakers seen and unseen during training. Code and pre-trained models will be made publicly available at https://github.com/NVIDIA/flowtron

Motivation & Objective

• Motivate the need for controllable non-textual information (prosody, style) in TTS beyond text-to-speech alone.
• Develop a generative model that learns an invertible mapping between mel-spectrograms and a latent z-space using normalizing flows.
• Enable manipulation of speech characteristics (pitch, tone, cadence, accent) via latent-space sampling and conditioning on text and speaker information.
• Achieve stable training by likelihood-based optimization without additional prenets/postnets or compound losses.
• Demonstrate expressive speech synthesis with variation control, interpolation, and style transfer for seen and unseen speakers.

Proposed method

• Adopt an autoregressive flow model where mel-spectrogram frames are generated via p(x) = product p(x_t|x_1:t-1) conditioned on text and speaker embeddings.
• Use invertible affine coupling layers to map a latent z (from Gaussian or Gaussian mixture) to mel-spectrograms, with log-determinant terms for likelihood training.
• Replace Tacotron components (Prenet/Postnet) and adopt a Tacotron-like text encoder with instance normalization and content-based attention.
• Parameterize z-space with a Gaussian or Gaussian mixture, optionally learned, enabling sampling and posterior conditioning on prior evidence for style control.
• Train by maximizing data likelihood; inference samples z from Gaussian or GM and inverts flows to generate mel-spectrograms, optionally using a posterior conditioned on prior samples.

Experimental results

Research questions

• RQ1 Can normalizing-flow-based TTS match state-of-the-art speech quality while providing controllable variability?
• RQ2 How does latent-space manipulation enable control over prosody, cadence, and style in TTS outputs?
• RQ3 Can Flowtron perform interpolation between samples and style transfer between seen/unseen speakers without expressive data labels?

Key findings

• Flowtron achieves MOS comparable to state-of-the-art TTS models, closing the quality gap with Tacotron 2 on mel-spectrogram synthesis.
• Sampling z from a Gaussian with varying variance (sigma^2) controls speech variation and duration, producing more expressive outputs without sacrificing quality.
• Flowtron supports interpolation in latent space to blend speaker characteristics and speech styles, including between seen speakers and unseen data.
• Posterior sampling based on prior evidence enables style transfer and expressive adjustments for both seen and unseen speakers.
• Gaussian Mixture Flowtron variants reveal disentangled components correlated with speaker gender and style attributes, enabling dimension-wise control.