[Paper Review] Emotional End-to-End Neural Speech Synthesizer

Younggun Lee, Azam Rabiee+1 more|arXiv (Cornell University)|Nov 15, 2017

Speech Recognition and Synthesis8 references58 citations

TL;DR

The paper extends Tacotron with emotion embeddings and training tricks to synthesize emotional speech, addressing exposure bias and attention alignment to improve quality.

ABSTRACT

In this paper, we introduce an emotional speech synthesizer based on the recent end-to-end neural model, named Tacotron. Despite its benefits, we found that the original Tacotron suffers from the exposure bias problem and irregularity of the attention alignment. Later, we address the problem by utilization of context vector and residual connection at recurrent neural networks (RNNs). Our experiments showed that the model could successfully train and generate speech for given emotion labels.

Motivation & Objective

• Introduce an emotion-conditioned end-to-end TTS model based on Tacotron.
• Address exposure bias and attention misalignment to improve long-frame speech generation.
• Enable generation of speech with multiple predefined emotions using learned emotion embeddings.

Proposed method

• Inject a learned emotion embedding into Tacotron's attention and decoder RNNs.
• Apply monotonic attention to enforce monotonic alignment during decoding.
• Use semi-teacher-forced training by mixing ground-truth and predicted frames as decoder input.
• Incorporate a context vector c_{t-1} into the attention RNN input to guide alignment.
• Add a residual connection in the CBHG text encoder to preserve current-step information and sharpen alignments.

Experimental results

Research questions

• RQ1 Can an emotion embedding integrated into Tacotron generate emotionally tinted speech?
• RQ2 Does enforcing monotonic attention and semi-teacher-forced training improve attention alignment and speech quality?
• RQ3 Do context vectors and CBHG residual connections improve alignment stability and intelligibility for emotional TTS?

Key findings

• The emotional Tacotron can generate speech carrying specified emotion labels.
• Monotonic attention leads to cleaner, more stable attention alignments than the original Tacotron.
• Semi-teacher-forced training helps mitigate exposure bias and reduces error accumulation.
• Incorporating a context vector and residual CBHG connections yields sharper, more reliable attention alignments.
• The model was trained on a Korean dataset with six emotion categories and about 21 hours of speech (text, audio, emotion label).
• The authors observed a correlation between attention alignment sharpness and speech quality, and provided qualitative results and GitHub resources for generated speech.