[Paper Review] ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech

Wei Ping, Kainan Peng+1 more|arXiv (Cornell University)|Jul 19, 2018

Speech Recognition and Synthesis37 references63 citations

TL;DR

The paper introduces a Gaussian inverse autoregressive flow (IAF) distillation from an autoregressive WaveNet to enable parallel waveform generation, and presents a fully convolutional end-to-end text-to-wave TTS architecture with a distilled parallel vocoder.

ABSTRACT

In this work, we propose a new solution for parallel wave generation by WaveNet. In contrast to parallel WaveNet (van den Oord et al., 2018), we distill a Gaussian inverse autoregressive flow from the autoregressive WaveNet by minimizing a regularized KL divergence between their highly-peaked output distributions. Our method computes the KL divergence in closed-form, which simplifies the training algorithm and provides very efficient distillation. In addition, we introduce the first text-to-wave neural architecture for speech synthesis, which is fully convolutional and enables fast end-to-end training from scratch. It significantly outperforms the previous pipeline that connects a text-to-spectrogram model to a separately trained WaveNet (Ping et al., 2018). We also successfully distill a parallel waveform synthesizer conditioned on the hidden representation in this end-to-end model.

Motivation & Objective

• Motivate fast, high-fidelity parallel waveform generation for WaveNet-based TTS.
• Introduce a fully convolutional end-to-end text-to-wave architecture trained from scratch.
• Demonstrate that a single Gaussian output is sufficient for modeling raw waveforms.
• Develop a regularized KL divergence-based distillation method to train Gaussian IAF from autoregressive WaveNet.
• Show that distilling a parallel vocoder conditioned on end-to-end hidden representations yields competitive results.

Proposed method

• Use a Gaussian autoregressive WaveNet as the teacher model with p(x_t|x_<t) = N(mu, sigma).
• Distill a Gaussian IAF as the student by minimizing a regularized KL divergence KL^reg between q(x_t|z_<t) and p(x_t|x_<t) with a closed-form per-time-step KL expression.
• Compute a combined loss including the regularized KL and an STFT-based frame loss to stabilize training.
• Develop a fully convolutional text-to-wave architecture (encoder, decoder, bridge-net, vocoder) that can be trained end-to-end from scratch, conditioning WaveNet on hidden representations rather than spectrograms.
• Share conditioning between teacher and student; optionally distill a parallel vocoder conditioned on end-to-end representations.
• Clip log-sigma during training for numerical stability and use a simple Gaussian output instead of mixtures.

Experimental results

Research questions

• RQ1 Can a single Gaussian output suffice for modeling raw WaveNet waveforms without quality loss?
• RQ2 Can a Gaussian IAF be distilled from an autoregressive WaveNet with a closed-form, stable KL divergence?
• RQ3 Does an end-to-end text-to-wave architecture trained from scratch outperform traditional pipeline approaches that separate text-to-spectrogram and vocoder?
• RQ4 How does a distilled parallel vocoder compare to an autoregressive vocoder in end-to-end TTS?
• RQ5 What are effective stabilization techniques (e.g., KL regularization, STFT loss) for this distillation process?

Key findings

• A single Gaussian output distribution suffices for WaveNet waveform modeling with comparable MOS to MoG/MoL/Softmax.
• Gaussian IAF distillation with regularized KL divergence achieves MOS around 4.16–4.22 for Student-1/Student-2, outperforming some baselines.
• End-to-end text-to-wave model outperforms the separately trained pipeline (MOS about 4.15 vs 3.81–3.73 for DV3+WaveNet baselines).
• Inference with the distilled parallel vocoder runs about 20x faster than real time on a GTX 1080 Ti.
• Distilled parallel vocoder is competitive with autoregressive vocoders, though slightly lower in MOS when conditioned only on end-to-end representations.
• The approach enables fast, fully convolutional, end-to-end TTS that trains from scratch and benefits from closed-form KL distillation.