[Paper Review] Robust and fine-grained prosody control of end-to-end speech synthesis

Younggun Lee, Taesu Kim|arXiv (Cornell University)|Nov 6, 2018

Speech Recognition and Synthesis12 references4 citations

TL;DR

This paper proposes temporally structured prosody embedding networks for end-to-end speech synthesis, enabling fine-grained, frame-level and phoneme-level control of pitch and amplitude. By learning variable-length prosody embeddings from speech or text and applying temporal normalization, the method achieves robust inter-speaker prosody transfer, including singing voice conversion, with a 29.4% relative improvement in MCD over GST Tacotron.

ABSTRACT

We propose prosody embeddings for emotional and expressive speech synthesis networks. The proposed methods introduce temporal structures in the embedding networks, thus enabling fine-grained control of the speaking style of the synthesized speech. The temporal structures can be designed either on the speech side or the text side, leading to different control resolutions in time. The prosody embedding networks are plugged into end-to-end speech synthesis networks and trained without any other supervision except for the target speech for synthesizing. It is demonstrated that the prosody embedding networks learned to extract prosodic features. By adjusting the learned prosody features, we could change the pitch and amplitude of the synthesized speech both at the frame level and the phoneme level. We also introduce the temporal normalization of prosody embeddings, which shows better robustness against speaker perturbations during prosody transfer tasks.

Motivation & Objective

• To enable fine-grained, frame-level and phoneme-level prosody control in end-to-end speech synthesis, overcoming limitations of fixed-length prosody embeddings.
• To address the robustness issue in inter-speaker prosody transfer when pitch ranges differ significantly between source and target speakers.
• To improve the consistency and explicitness of prosody control by learning temporal structures in prosody embeddings.
• To demonstrate effective prosody transfer on singing voices, where pitch and timing are highly sensitive.
• To show that temporal normalization of prosody embeddings enhances robustness during prosody transfer tasks.

Proposed method

• Introduces two types of variable-length prosody embeddings: one derived from the speech signal (speech-side) and one from the text input (text-side), both matching the sequence length of the input.
• Uses GRU-based networks to generate temporal prosody embeddings that preserve sequential prosodic information across time.
• Applies temporal normalization to prosody embeddings during training to reduce sensitivity to speaker-specific pitch variations.
• Plugs the prosody embedding networks into an end-to-end TTS model (simplified Tacotron) without additional supervision beyond the target speech.
• Employs a content-based attention mechanism to compute style token contributions, enabling explicit prosody control via embedding manipulation.
• Uses Mean Cepstral Distortion (MCD) with 13 MFCCs as the primary evaluation metric for prosody reconstruction and transfer quality.

Experimental results

Research questions

• RQ1 Can variable-length prosody embeddings enable fine-grained, frame-level control of pitch and amplitude in synthesized speech?
• RQ2 Does temporal normalization of prosody embeddings improve robustness in inter-speaker prosody transfer, especially when pitch ranges differ?
• RQ3 Can the proposed method successfully transfer prosody from a singing voice to a different speaker while preserving melody and rhythm?
• RQ4 How does the performance of the proposed prosody embedding methods compare to GST Tacotron in terms of prosody reconstruction accuracy?
• RQ5 Can explicit prosody control be achieved by manipulating prosody embedding values at the phoneme or frame level?

Key findings

• The speech-side prosody control method achieved the lowest MCD of 0.294, significantly outperforming GST Tacotron’s MCD of 0.413.
• Temporal normalization reduced the MCD for female-to-male prosody transfer from 0.531 to 0.518, demonstrating improved robustness.
• Fixed prosody embeddings across all frames produced flat, consistent speech, confirming explicit and controllable prosody manipulation.
• The proposed method successfully reconstructed the melody of a song using prosody embeddings extracted from an unseen singer, while GST Tacotron failed to do so.
• Speech-side prosody control generated audio nearly identical to the original song, indicating high-fidelity prosody transfer in singing tasks.
• The method enabled consistent prosody control at both frame-level and phoneme-level, with no unintended prosodic variation across identical phonemes.