[Paper Review] MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesis

Yi Lei, Shan Yang+2 more|arXiv (Cornell University)|Jan 17, 2022

Speech Recognition and Synthesis4 citations

TL;DR

MsEmoTTS proposes a multi-scale emotional speech synthesis framework that models emotion at global (category), utterance (prosody pattern), and local (syllable-level strength) levels using three dedicated modules. It enables flexible emotional speech synthesis via reference audio transfer, text-based emotion prediction, and manual emotion strength control, outperforming prior methods in both emotion transfer and prediction tasks on a Chinese corpus.

ABSTRACT

Expressive synthetic speech is essential for many human-computer interaction and audio broadcast scenarios, and thus synthesizing expressive speech has attracted much attention in recent years. Previous methods performed the expressive speech synthesis either with explicit labels or with a fixed-length style embedding extracted from reference audio, both of which can only learn an average style and thus ignores the multi-scale nature of speech prosody. In this paper, we propose MsEmoTTS, a multi-scale emotional speech synthesis framework, to model the emotion from different levels. Specifically, the proposed method is a typical attention-based sequence-to-sequence model and with proposed three modules, including global-level emotion presenting module (GM), utterance-level emotion presenting module (UM), and local-level emotion presenting module (LM), to model the global emotion category, utterance-level emotion variation, and syllable-level emotion strength, respectively. In addition to modeling the emotion from different levels, the proposed method also allows us to synthesize emotional speech in different ways, i.e., transferring the emotion from reference audio, predicting the emotion from input text, and controlling the emotion strength manually. Extensive experiments conducted on a Chinese emotional speech corpus demonstrate that the proposed method outperforms the compared reference audio-based and text-based emotional speech synthesis methods on the emotion transfer speech synthesis and text-based emotion prediction speech synthesis respectively. Besides, the experiments also show that the proposed method can control the emotion expressions flexibly. Detailed analysis shows the effectiveness of each module and the good design of the proposed method.

Motivation & Objective

• To address the limitation of single-scale emotion modeling in existing expressive TTS systems, which fail to capture the multi-granularity nature of human prosody.
• To enable flexible emotional speech synthesis through three distinct modes: reference-based emotion transfer, text-based emotion prediction, and manual emotion strength control.
• To model emotion at three hierarchical levels: global emotion category, utterance-level prosodic variation, and syllable-level emotion intensity.
• To improve the expressiveness and diversity of synthesized speech by capturing fine-grained emotional variations across different speech units.

Proposed method

• The model uses an attention-based sequence-to-sequence architecture with three specialized modules: Global-level emotion presenting module (GM), Utterance-level emotion presenting module (UM), and Local-level emotion presenting module (LM).
• GM classifies the overall emotion category of the utterance using a pre-trained emotion classifier, providing a global style signal.
• UM learns the prosodic pattern (e.g., intonation contour) within the utterance by modeling the variation of pitch and energy across the utterance.
• LM assigns emotion strength values at the syllable level to control localized emotional intensity, with experiments showing syllables are more suitable than phonemes in Chinese.
• Emotion representations from reference audio or text are integrated into the TTS decoder via adaptive layer normalization, enabling style transfer and control.
• The model supports end-to-end training with multi-task learning, where emotion classification and strength prediction are jointly optimized.

Experimental results

Research questions

• RQ1 Can modeling emotion at multiple scales—global, utterance, and local—improve the expressiveness and naturalness of synthesized emotional speech?
• RQ2 How does the proposed multi-scale emotion representation compare to single-scale reference-based or label-based methods in emotion transfer and prediction tasks?
• RQ3 Does using syllable-level emotion strength improve synthesis quality compared to phoneme-level or sentence-level representations in Chinese speech?
• RQ4 To what extent can the model support flexible emotional control through manual input of emotion category and strength?

Key findings

• MsEmoTTS outperforms reference-based and text-based emotional TTS methods in both emotion transfer and emotion prediction tasks on a Chinese emotional speech corpus.
• The model achieves a Mel-cepstral distortion (MCD) of 4.11 dB when using syllable-level emotion strengths, significantly lower than the 4.65 dB achieved with sentence-level strengths, confirming the benefit of fine-grained local modeling.
• The syllable-level emotion strength method outperforms the phoneme-level method by 1.1% in MCD reduction, indicating better suitability for tonal languages like Chinese.
• Visual analysis of mel-spectrograms and f0 contours confirms that the utterance-level module (UM) successfully captures and transfers pitch variation trends from reference audio.
• Manual control of global emotion category and local emotion strength enables precise synthesis of desired emotional expressions, demonstrating the model’s flexibility.
• When conflicting references are used for GM and UM (e.g., different emotion categories), the synthesized speech exhibits unnatural hybrid emotions, indicating interdependence between global and utterance-level prosody.