[Paper Review] Limited Data Emotional Voice Conversion Leveraging Text-to-Speech: Two-stage Sequence-to-Sequence Training

Kun Zhou, Berrak Şişman+1 more|arXiv (Cornell University)|Mar 31, 2021

Speech Recognition and Synthesis38 references4 citations

TL;DR

This paper proposes a two-stage sequence-to-sequence (seq2seq) framework for emotional voice conversion (EVC) that leverages text-to-speech (TTS) pretraining to overcome data scarcity. By first initializing style disentanglement via multi-speaker TTS and then fine-tuning on limited emotional speech data, the method achieves state-of-the-art performance in both spectral and prosodic conversion, significantly outperforming baselines in objective and subjective evaluations with minimal labeled data.

ABSTRACT

Emotional voice conversion (EVC) aims to change the emotional state of an utterance while preserving the linguistic content and speaker identity. In this paper, we propose a novel 2-stage training strategy for sequence-to-sequence emotional voice conversion with a limited amount of emotional speech data. We note that the proposed EVC framework leverages text-to-speech (TTS) as they share a common goal that is to generate high-quality expressive voice. In stage 1, we perform style initialization with a multi-speaker TTS corpus, to disentangle speaking style and linguistic content. In stage 2, we perform emotion training with a limited amount of emotional speech data, to learn how to disentangle emotional style and linguistic information from the speech. The proposed framework can perform both spectrum and prosody conversion and achieves significant improvement over the state-of-the-art baselines in both objective and subjective evaluation.

Motivation & Objective

• To address the challenge of training high-quality emotional voice conversion (EVC) systems with limited emotional speech data.
• To leverage the shared goal of expressive speech generation between text-to-speech (TTS) and EVC to improve model generalization and disentanglement.
• To enable many-to-many emotional voice conversion without requiring parallel training data.
• To improve prosody and duration modeling by shifting from frame-wise to sequence-level alignment via attention mechanisms.
• To enhance vocoder quality through emotion-specific fine-tuning of WaveRNN.

Proposed method

• Stage 1: Perform style initialization using a multi-speaker TTS corpus to disentangle linguistic content from speaking style.
• Stage 2: Fine-tune the entire model on limited emotional speech data to learn emotion-specific style disentanglement.
• Integrate an emotion encoder and classifier to suppress emotion-related information in the linguistic space.
• Use an attention-based seq2seq architecture to model alignment between acoustic and linguistic embeddings for hierarchical prosody modeling.
• Apply emotion-specific fine-tuning to the WaveRNN vocoder using only limited emotional speech data to improve perceptual quality.
• Train the model end-to-end with joint optimization of spectral and duration conversion using sequence-level supervision.

Figure 1: Visualization of emotion embeddings derived from (a) style encoder and (b) emotion encoder. Each point represents the emotion embedding of a reference utterance.

Experimental results

Research questions

• RQ1 Can a two-stage training strategy reduce data dependency in seq2seq emotional voice conversion?
• RQ2 Can TTS pretraining effectively disentangle linguistic content from emotional and speaker styles in low-data regimes?
• RQ3 Does sequence-level attention modeling improve prosody and duration conversion compared to frame-wise methods?
• RQ4 Can emotion-specific fine-tuning of a WaveRNN vocoder enhance perceptual quality with minimal data?
• RQ5 How does the proposed framework compare to SOTA baselines in both objective metrics and subjective listening tests?

Key findings

• The proposed Seq2seq-EVC-WA1 and Seq2seq-EVC-WA2 models achieve the lowest MCD values across all emotion pairs, outperforming CycleGAN-EVC and StarGAN-EVC.
• The models with fine-tuned WaveRNN (Seq2seq-EVC-WA2) achieve the best DDUR results, demonstrating superior duration conversion capability.
• Subjective listening tests show that Seq2seq-EVC-WA2 is ranked as the best in speech quality by 90% of listeners for the Neu-Sad emotion pair.
• Emotion similarity scores indicate that Seq2seq-EVC-WA2 significantly outperforms baselines, especially for the Neu-Sur pair, with a mean score of +1.8 on the -2 to +2 scale.
• The attention mechanism enables variable phonetic duration mapping, as shown in alignment visualization where the same source utterance produces different durations for happy and sad targets.
• Emotional fine-tuning of the WaveRNN vocoder leads to consistent improvements in speech quality, as confirmed by best-worst scaling (BWS) results.

Figure 2: The proposed 2-stage training strategy for seq2seq emotional voice conversion with limited emotional speech data.