[Paper Review] Tacotron: Towards End-to-End Speech Synthesis

Yuxuan Wang, RJ Skerry-Ryan+12 more|arXiv (Cornell University)|Mar 29, 2017

Speech Recognition and SynthesisComputer Science20 references151 citations

TL;DR

Tacotron proposes an end-to-end seq2seq TTS model that learns from <text, audio> pairs to generate spectrograms from characters, achieving higher naturalness than a production parametric system while enabling faster frame-based inference.

ABSTRACT

A text-to-speech synthesis system typically consists of multiple stages, such as a text analysis frontend, an acoustic model and an audio synthesis module. Building these components often requires extensive domain expertise and may contain brittle design choices. In this paper, we present Tacotron, an end-to-end generative text-to-speech model that synthesizes speech directly from characters. Given pairs, the model can be trained completely from scratch with random initialization. We present several key techniques to make the sequence-to-sequence framework perform well for this challenging task. Tacotron achieves a 3.82 subjective 5-scale mean opinion score on US English, outperforming a production parametric system in terms of naturalness. In addition, since Tacotron generates speech at the frame level, it's substantially faster than sample-level autoregressive methods.

Motivation & Objective

• Eliminate hand-engineered text and feature processing in TTS by using an end-to-end model trained on <text, audio> pairs.
• Enable conditioning on attributes such as speaker or language through end-to-end training.
• Improve alignment and prosody handling for longer and diverse utterances.
• Increase synthesis speed by generating frame-based spectrograms rather than sample-by-sample outputs.

Proposed method

• Use a sequence-to-sequence architecture with attention to map character sequences to mel-scale spectrogram frames.
• Introduce the CBHG (Convolution Bank + Highway Network + Bidirectional GRU) module to robustly encode text sequences.
• Predict multiple frames per decoder step (reduction factor r) to accelerate convergence and training.
• Apply a post-processing CBHG to convert seq2seq targets to higher-quality spectrograms for waveform synthesis via Griffin-Lim.
• Train with L1 losses on seq2seq mel-spectrogram and post-processing linear-spectrogram targets.
• Synthesize audio with Griffin-Lim after predicting spectrograms; apply 1.2 power scaling to magnitudes before Griffin-Lim to reduce artifacts.

Experimental results

Research questions

• RQ1 Can an end-to-end model trained on raw text achieve natural-sounding speech without hand-engineered Linguistic features or HMM aligners?
• RQ2 Does a CBHG-based encoder improve alignment and generalization for character-level inputs in TTS?
• RQ3 Does predicting multiple frames per decoder step (reduction factor) speed up convergence and improve alignment stability?
• RQ4 What is the impact of a post-processing net on the quality of synthesized speech when using Griffin-Lim for waveform reconstruction?

Key findings

• Tacotron achieves a 3.82 mean opinion score (MOS) on US English, outperforming a production parametric system in naturalness.
• The model runs frame-based and is substantially faster than sample-level autoregressive methods.
• A CBHG encoder reduces mispronunciations and overfitting compared to a vanilla GRU encoder.
• A post-processing net improves harmonic detail and high-frequency formants, reducing synthesis artifacts.
• Griffin-Lim-based waveform synthesis with spectral magnitudes raised to 1.2 improves perceptual quality.
• On internal data (about 24.6 hours) using a single female speaker, Tacotron demonstrates strong naturalness and robustness.