[Paper Review] FastSpeech 2: Fast and High-Quality End-to-End Text to Speech

Yi Ren|arXiv (Cornell University)|Jun 8, 2020

Speech Recognition and Synthesis40 references513 citations

TL;DR

FastSpeech 2 directly trains on ground-truth mel-spectrograms and adds pitch and energy variance information to ease one-to-many mapping; FastSpeech 2s further generates waveforms directly in parallel for faster end-to-end TTS.

ABSTRACT

Non-autoregressive text to speech (TTS) models such as FastSpeech can synthesize speech significantly faster than previous autoregressive models with comparable quality. The training of FastSpeech model relies on an autoregressive teacher model for duration prediction (to provide more information as input) and knowledge distillation (to simplify the data distribution in output), which can ease the one-to-many mapping problem (i.e., multiple speech variations correspond to the same text) in TTS. However, FastSpeech has several disadvantages: 1) the teacher-student distillation pipeline is complicated and time-consuming, 2) the duration extracted from the teacher model is not accurate enough, and the target mel-spectrograms distilled from teacher model suffer from information loss due to data simplification, both of which limit the voice quality. In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs. Specifically, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs in training and use predicted values in inference. We further design FastSpeech 2s, which is the first attempt to directly generate speech waveform from text in parallel, enjoying the benefit of fully end-to-end inference. Experimental results show that 1) FastSpeech 2 achieves a 3x training speed-up over FastSpeech, and FastSpeech 2s enjoys even faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models. Audio samples are available at https://speechresearch.github.io/fastspeech2/.

Motivation & Objective

• Address the one-to-many mapping problem in non-autoregressive TTS by enriching input with variance information (duration, pitch, energy).
• Simplify the training pipeline by directly using ground-truth mel-spectrograms instead of teacher-distilled targets.
• Improve duration accuracy and introduce pitch and energy predictors to enhance voice quality.
• Develop FastSpeech 2s to enable fully end-to-end text-to-waveform synthesis with fast inference.

Proposed method

• Encoder converts phoneme embeddings into hidden representations.
• Variance adaptor adds duration, pitch, and energy information to the hidden sequence.
• Duration predictor uses forced alignment (MFA) targets for accurate phoneme durations.
• Pitch predictor predicts pitch in the frequency domain via continuous wavelet transform (CWT) and uses iCWT for inference.
• Energy predictor predicts frame-level energy and adds its embedding to the hidden sequence.
• Mel-spectrogram decoder generates mel-spectrograms in parallel; training targets are ground-truth mel-spectrograms (no teacher-student distillation).
• FastSpeech 2s extends FastSpeech 2 by directly generating waveform from text using a WaveNet-like decoder with adversarial training and a parallel vocoder-based discriminator; enables end-to-end waveform synthesis.

Experimental results

Research questions

• RQ1 Can removing teacher-student distillation improve voice quality by training directly on ground-truth mel-spectrograms?
• RQ2 Does incorporating explicit duration, pitch (via CWT), and energy as variance information improve the naturalness of synthesized speech?
• RQ3 How does predicting pitch in the frequency domain affect prosody and MOS?
• RQ4 Can a fully end-to-end non-autoregressive TTS model generate waveform directly with FastSpeech 2s while retaining quality and speed?

Key findings

• FastSpeech 2 achieves 3x training speed-up over FastSpeech.
• FastSpeech 2 and 2s outperform FastSpeech in voice quality; FastSpeech 2 can even surpass autoregressive models in MOS.
• FastSpeech 2 and 2s provide more accurate pitch and energy, improving prosody and CMOS scores compared to FastSpeech.
• Pitch is predicted in the frequency domain using CWT, improving pitch modeling and CMOS scores over time-domain prediction.
• FastSpeech 2s enables fully end-to-end waveform generation with faster inference than cascaded systems.
• FastSpeech 2 provides substantial inference-time speedups (RTF around 1.95e-2) and FastSpeech 2s further improves endpoint latency.