[Paper Review] Review of end-to-end speech synthesis technology based on deep learning

Zhaoxi Mu, Xinyu Yang+1 more|arXiv (Cornell University)|Apr 20, 2021

Speech Recognition and Synthesis192 references20 citations

TL;DR

This paper reviews end-to-end deep learning-based text-to-speech (TTS) systems, analyzing their three core components—text front-end, acoustic model, and vocoder—while comparing methods by design focus. It highlights state-of-the-art performance close to human-like speech and identifies future directions in fine-grained style control, unsupervised representation learning, fully end-to-end training, and cross-task transfer from image generation and ASR/NMT.

ABSTRACT

As an indispensable part of modern human-computer interaction system, speech synthesis technology helps users get the output of intelligent machine more easily and intuitively, thus has attracted more and more attention. Due to the limitations of high complexity and low efficiency of traditional speech synthesis technology, the current research focus is the deep learning-based end-to-end speech synthesis technology, which has more powerful modeling ability and a simpler pipeline. It mainly consists of three modules: text front-end, acoustic model, and vocoder. This paper reviews the research status of these three parts, and classifies and compares various methods according to their emphasis. Moreover, this paper also summarizes the open-source speech corpus of English, Chinese and other languages that can be used for speech synthesis tasks, and introduces some commonly used subjective and objective speech quality evaluation method. Finally, some attractive future research directions are pointed out.

Motivation & Objective

• To provide a systematic review of end-to-end deep learning-based TTS technology, focusing on recent advances in modeling, training, and inference.
• To analyze and compare methods across the three main components of TTS: text front-end, acoustic model, and vocoder, based on their design and performance.
• To summarize publicly available multilingual speech corpora and standard evaluation metrics for TTS research.
• To identify key challenges and future research directions in TTS, including style control, representation learning, and fully end-to-end frameworks.
• To explore the potential of transferring techniques from related fields such as image generation, machine translation, and automatic speech recognition to improve TTS performance and data efficiency.

Proposed method

• Classifies and compares TTS methods based on their emphasis in text front-end (e.g., phoneme-based, subword, or end-to-end tokenization), acoustic modeling (e.g., Tacotron, FastSpeech, ClariNet), and vocoder design (e.g., WaveNet, WaveGlow, WaveNet-based autoregressive or flow-based models).
• Reviews the use of deep neural networks—particularly RNNs, Transformers, and self-attention mechanisms—for modeling long-range dependencies in speech sequences.
• Analyzes the role of intermediate representations such as mel-spectrograms, raw waveforms, and linguistic features as conditioning inputs to the vocoder.
• Examines autoregressive and flow-based generative models in vocoders, including normalizing flows and adversarial training for high-fidelity speech generation.
• Discusses the integration of multi-task learning and knowledge distillation by combining TTS with ASR, speaker embedding, and emotion recognition models.
• Proposes the use of unsupervised representation learning and meta-learning to reduce dependency on large-scale annotated speech-text pairs, especially for low-resource languages.

Experimental results

Research questions

• RQ1 How do different architectures in the text front-end, acoustic model, and vocoder contribute to the quality and efficiency of end-to-end TTS systems?
• RQ2 What are the key limitations of current TTS systems in modeling fine-grained speech styles such as emotion, intonation, and rhythm at the word or phrase level?
• RQ3 To what extent can unsupervised or self-supervised representation learning reduce the need for large-scale, manually transcribed speech-text datasets?
• RQ4 Can fully end-to-end TTS models that map text directly to raw waveforms outperform modular pipelines with intermediate acoustic features?
• RQ5 How can techniques from image generation, machine translation, and automatic speech recognition be adapted to improve TTS performance and data efficiency?

Key findings

• State-of-the-art end-to-end TTS models based on deep learning can synthesize speech that is nearly indistinguishable from human speech in terms of naturalness and intelligibility.
• Models such as Tacotron 2 and WaveNet achieve high perceptual quality, with MOS (Mean Opinion Score) values approaching human-level performance on standard benchmarks.
• The integration of attention mechanisms and residual connections in Transformer-based TTS models improves alignment and reduces training instability compared to RNN-based models.
• Vocoder components based on normalizing flows (e.g., WaveGlow) and autoregressive models (e.g., WaveNet) significantly improve speech quality while maintaining high inference speed.
• Despite progress, current systems still struggle with precise, fine-grained control over speech style, particularly emotion and prosody, due to limited and poorly annotated training data.
• The lack of sufficient paired text-speech data for low-resource languages remains a major barrier, with most SOTA systems limited to English and Mandarin.