[Paper Review] FastSpeech: Fast, Robust and Controllable Text to Speech

Yi Ren, Yangjun Ruan+5 more|arXiv (Cornell University)|May 22, 2019

Speech Recognition and Synthesis26 references581 citations

TL;DR

FastSpeech introduces a non-autoregressive Transformer-based TTS that generates mel-spectrograms in parallel using a length regulator and a duration predictor, achieving major speedups and improved robustness with controllable voice speed.

ABSTRACT

Neural network based end-to-end text to speech (TTS) has significantly improved the quality of synthesized speech. Prominent methods (e.g., Tacotron 2) usually first generate mel-spectrogram from text, and then synthesize speech from the mel-spectrogram using vocoder such as WaveNet. Compared with traditional concatenative and statistical parametric approaches, neural network based end-to-end models suffer from slow inference speed, and the synthesized speech is usually not robust (i.e., some words are skipped or repeated) and lack of controllability (voice speed or prosody control). In this work, we propose a novel feed-forward network based on Transformer to generate mel-spectrogram in parallel for TTS. Specifically, we extract attention alignments from an encoder-decoder based teacher model for phoneme duration prediction, which is used by a length regulator to expand the source phoneme sequence to match the length of the target mel-spectrogram sequence for parallel mel-spectrogram generation. Experiments on the LJSpeech dataset show that our parallel model matches autoregressive models in terms of speech quality, nearly eliminates the problem of word skipping and repeating in particularly hard cases, and can adjust voice speed smoothly. Most importantly, compared with autoregressive Transformer TTS, our model speeds up mel-spectrogram generation by 270x and the end-to-end speech synthesis by 38x. Therefore, we call our model FastSpeech.

Motivation & Objective

• Address slow inference, robustness issues (word skipping/repeating), and lack of controllability in autoregressive TTS models.
• Propose a parallel mel-spectrogram generation framework based on a feed-forward Transformer (FFT).
• Leverage phoneme duration alignment via a duration predictor and a length regulator to match mel-spectrogram length.
• Enable controllable speech synthesis by adjusting phoneme durations to change speed and prosody.

Proposed method

• Use a Feed-Forward Transformer (FFT) with self-attention and 1D convolution blocks for phoneme-to-mel-spectrogram transformation.
• Introduce a Length Regulator that up-samples phoneme representations according to predicted phoneme durations to match mel-spectrogram length.
• Predict phoneme durations with a duration predictor trained from an autoregressive teacher model, using a diagonal-attention alignment to derive ground-truth durations.
• Train FastSpeech using sequence-level knowledge distillation from an autoregressive Transformer TTS model (teacher) to a parallel (student) model.
• Apply a WaveGlow vocoder for end-to-end audio synthesis from generated mel-spectrograms.

Experimental results

Research questions

• RQ1 Can non-autoregressive, parallel generation of mel-spectrograms achieve comparable speech quality to autoregressive models?
• RQ2 Does the length regulator and accurate phoneme duration prediction reduce word skipping and repeating errors?
• RQ3 How much speedup is achievable in mel-spectrogram generation and end-to-end synthesis compared to autoregressive TTS?
• RQ4 To what extent can voice speed and prosody be controllably adjusted via phoneme durations?

Key findings

• FastSpeech nearly matches autoregressive Transformer TTS in speech quality (MOS close to Transformer TTS).
• Mel-spectrogram generation is sped up by 269.4x and end-to-end synthesis by 38.3x compared to autoregressive Transformer TTS.
• FastSpeech virtually eliminates word skipping and repeating on hard test sentences (0% error).
• Voice speed can be smoothly adjusted from 0.5x to 1.5x via phoneme duration regulation.
• Adding breaks between words can improve prosody through duration control.
• Ablations show 1D convolution and sequence-level knowledge distillation contribute positively to performance.