[Paper Review] Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech

Jaehyeon Kim, Jungil Kong+1 more|arXiv (Cornell University)|Jun 11, 2021

Speech Recognition and SynthesisComputer Science38 references121 citations

TL;DR

VITS presents a parallel end-to-end TTS model using a conditional VAE with normalizing flows and adversarial training to generate natural speech, plus a stochastic duration predictor for diverse rhythms, achieving MOS close to ground truth on LJ Speech and strong multi-speaker performance on VCTK.

ABSTRACT

Several recent end-to-end text-to-speech (TTS) models enabling single-stage training and parallel sampling have been proposed, but their sample quality does not match that of two-stage TTS systems. In this work, we present a parallel end-to-end TTS method that generates more natural sounding audio than current two-stage models. Our method adopts variational inference augmented with normalizing flows and an adversarial training process, which improves the expressive power of generative modeling. We also propose a stochastic duration predictor to synthesize speech with diverse rhythms from input text. With the uncertainty modeling over latent variables and the stochastic duration predictor, our method expresses the natural one-to-many relationship in which a text input can be spoken in multiple ways with different pitches and rhythms. A subjective human evaluation (mean opinion score, or MOS) on the LJ Speech, a single speaker dataset, shows that our method outperforms the best publicly available TTS systems and achieves a MOS comparable to ground truth.

Motivation & Objective

• Bridge two-stage TTS limitations by enabling end-to-end training with latent variable modeling.
• Increase expressive power of the prior and posterior distributions via normalizing flows.
• Model and utilize a stochastic duration predictor to capture diverse rhythms in speech.
• Leverage adversarial training to enhance waveform realism beyond mel-spectrogram reconstructions.
• Demonstrate superior quality and multi-speaker capabilities compared to public two-stage systems.

Proposed method

• Formulates TTS as a conditional VAE with prior p(z|c) enhanced by a normalizing flow f_theta for expressive latent space.
• Uses a posterior encoder q_phi(z|x_lin) and reconstruction in mel-spectrogram domain with L1 loss.
• Estimates text–speech alignment A via Monotonic Alignment Search (MAS) adapted to maximize ELBO.
• Introduces a stochastic duration predictor based on variational dequantization and variational data augmentation to model speech rhythm.
• Incorporates adversarial training with a HiFi-GAN-like decoder and a discriminator D, plus a feature-matching loss for stable, high-quality waveform generation.
• Trains with windowed generator training to improve efficiency while enabling end-to-end generation.

Experimental results

Research questions

• RQ1 Can a conditional VAE with a flow-based prior produce high-quality end-to-end waveform synthesis without intermediate representations?
• RQ2 Does MAS-based alignment estimation integrated into ELBO optimization yield better alignments for text-to-speech?
• RQ3 Can a stochastic duration predictor deliver diverse rhythms in parallel TTS while maintaining naturalness?
• RQ4 What is the impact of adversarial training and feature matching on end-to-end TTS synthesis quality?
• RQ5 How well does the proposed end-to-end model generalize to multi-speaker corpora?

Key findings

• VITS achieves MOS comparable to ground truth on LJ Speech, outperforming public two-stage systems.
• A normalizing flow in the prior encoder significantly improves MOS (1.52 MOS decrease when removed).
• Using linear-scale spectrogram as posterior input yields higher quality than mel-spectrogram input in the posterior path.
• On VCTK, VITS attains higher MOS than Tacotron 2+HiFi-GAN and Glow-TTS+HiFi-GAN baselines, demonstrating effective multi-speaker modeling.
• The stochastic duration predictor enables diverse phoneme durations and pitch, yielding varied speech rhythms while maintaining quality.
• VITS delivers faster synthesis than Glow-TTS+HiFi-GAN in end-to-end generation, achieving real-time or faster performance on GPUs.