[Paper Review] Towards Multi-Scale Style Control for Expressive Speech Synthesis

Xiang Li, Changhe Song+4 more|arXiv (Cornell University)|Apr 8, 2021

Speech Recognition and Synthesis17 references4 citations

TL;DR

This paper proposes a multi-scale reference encoder for end-to-end expressive TTS that jointly extracts global-style embeddings (GSE) and local-prosody embeddings (LPE) from reference speech, enabling simultaneous control over utterance-level emotion and fine-grained prosody. The method improves style transfer accuracy and robustness, especially in cross-emotion and non-parallel settings, outperforming mono-scale baselines in both objective and subjective evaluations.

ABSTRACT

This paper introduces a multi-scale speech style modeling method for end-to-end expressive speech synthesis. The proposed method employs a multi-scale reference encoder to extract both the global-scale utterance-level and the local-scale quasi-phoneme-level style features of the target speech, which are then fed into the speech synthesis model as an extension to the input phoneme sequence. During training time, the multi-scale style model could be jointly trained with the speech synthesis model in an end-to-end fashion. By applying the proposed method to style transfer task, experimental results indicate that the controllability of the multi-scale speech style model and the expressiveness of the synthesized speech are greatly improved. Moreover, by assigning different reference speeches to extraction of style on each scale, the flexibility of the proposed method is further revealed.

Motivation & Objective

• To address the limitation of existing expressive TTS models that focus on only one scale of speech style (global or local), leading to poor control over either utterance-level emotion or fine-grained prosody.
• To develop a unified, end-to-end trainable framework that models both global-scale (e.g., speaker timbre, emotion) and local-scale (e.g., pitch, energy, pause) style features simultaneously.
• To improve the controllability and expressiveness of synthesized speech by integrating multi-scale style features into the TTS backbone via a reference attention mechanism.
• To enable flexible style transfer using multiple reference speeches—e.g., combining global emotion from one reference and local prosody from another—without requiring parallel data.

Proposed method

• A multi-scale reference encoder is designed to extract both a global-style embedding (GSE) vector and a local-prosody embedding (LPE) sequence from the mel-spectrogram of a reference speech input.
• The reference encoder uses strided convolutions to downsample the input spectrogram, producing intermediate features with granularity closer to phonemes (quasi-phoneme-scale), improving alignment with phonetic units.
• The LPE sequence is aligned to the phoneme sequence using a reference attention mechanism, similar to content-based attention in attention-based TTS.
• The multi-scale reference encoder and reference attention are jointly trained with the TTS model in an end-to-end fashion, enabling joint optimization of style modeling and speech generation.
• The method supports non-parallel and cross-emotion style transfer by assigning different reference speeches for global and local style components.
• The model architecture is evaluated using a TTS backbone with an encoder-decoder structure, where style features are concatenated to the phoneme embeddings for conditional generation.

Experimental results

Research questions

• RQ1 Can a unified end-to-end TTS system effectively model both global and local speech style features without relying on auxiliary labels or separate training stages?
• RQ2 How does multi-scale style modeling improve the controllability of synthesized speech in terms of global emotion and fine-grained prosody?
• RQ3 To what extent does using quasi-phoneme-scale style features (as opposed to frame-scale) enhance the robustness and accuracy of reference attention and speech synthesis?
• RQ4 Can the model perform flexible style transfer by combining global and local style components from different reference speeches, even when they are non-parallel?
• RQ5 How does the proposed method compare to mono-scale baselines (e.g., global-only or local-only) in both objective and subjective evaluations?

Key findings

• The proposed multi-scale model significantly outperforms mono-scale baselines in both global emotion and local prosody transfer, as confirmed by Mean Opinion Score (MOS) evaluations.
• Subjective evaluation shows that the proposed model achieves higher similarity scores for both global emotion (mean MOS: 4.21) and local prosody (mean MOS: 4.15) compared to baselines.
• The model successfully performs cross-emotion style transfer, where the global emotion is transferred from one reference while preserving the local prosody from another, demonstrating high flexibility.
• The quasi-phoneme-scale LPE representation leads to more robust reference attention alignment, avoiding errors in synthesizing final phonemes—unlike the frame-scale baseline (Base-FS), which fails to generate the last character in some cases.
• In non-parallel style transfer, the model maintains high quality and coherence, indicating that it can generalize beyond parallel training data and handle diverse reference inputs effectively.
• The ablation study confirms that removing downsampling (i.e., reverting to frame-scale) degrades reference attention alignment and speech quality, validating the importance of quasi-phoneme-scale granularity.