[Paper Review] Emotion controllable speech synthesis using emotion-unlabeled dataset with the assistance of cross-domain speech emotion recognition

Xiong Cai, Dongyang Dai+4 more|arXiv (Cornell University)|Oct 26, 2020

Speech Recognition and Synthesis27 references4 citations

TL;DR

This paper proposes a novel emotion-controllable text-to-speech (TTS) system that trains on a fully emotion-unlabeled TTS dataset by leveraging a cross-domain speech emotion recognition (SER) model and an auxiliary emotion prediction task using global style tokens (GST). The method achieves high emotional expressiveness with minimal degradation in speech quality, demonstrating effectiveness in zero-shot emotional control without requiring manually labeled emotion data.

ABSTRACT

Neural text-to-speech (TTS) approaches generally require a huge number of high quality speech data, which makes it difficult to obtain such a dataset with extra emotion labels. In this paper, we propose a novel approach for emotional TTS synthesis on a TTS dataset without emotion labels. Specifically, our proposed method consists of a cross-domain speech emotion recognition (SER) model and an emotional TTS model. Firstly, we train the cross-domain SER model on both SER and TTS datasets. Then, we use emotion labels on the TTS dataset predicted by the trained SER model to build an auxiliary SER task and jointly train it with the TTS model. Experimental results show that our proposed method can generate speech with the specified emotional expressiveness and nearly no hindering on the speech quality.

Motivation & Objective

• To address the scarcity of emotion-annotated datasets in emotional TTS, which limits practical deployment.
• To enable emotion control in TTS without requiring manually labeled emotion data on the TTS dataset.
• To improve the modeling of emotion-related prosody in neural TTS using weak supervision from cross-domain SER.
• To maintain high speech quality while enabling precise emotional expressiveness through auxiliary training.
• To reduce dependency on expensive emotion annotations by leveraging external SER datasets and domain adaptation.

Proposed method

• A cross-domain SER model is pre-trained on a labeled SER dataset (source domain) and an unlabeled TTS dataset (target domain) using domain adaptation via MMD to reduce distribution shift.
• Soft emotion labels are predicted from the SER model's softmax output on the TTS dataset and used as weak supervision for an auxiliary emotion prediction task.
• The TTS model is jointly trained with the GST module and the auxiliary emotion prediction head, where the emotion prediction is based on the learned style token weights.
• A top-K reference selection scheme is used to form emotion-specific reference audio sets: for each emotion, the K most confidently predicted audios are averaged to generate the emotion embedding.
• The final TTS inference uses the averaged style token weights from the selected reference set to control the emotional prosody of synthesized speech.
• The method leverages the GST framework to disentangle content and style, with emotion modeling enhanced by the auxiliary prediction task.

Experimental results

Research questions

• RQ1 Can a TTS system achieve controllable emotional expressiveness when trained solely on an emotion-unlabeled dataset?
• RQ2 How effective is cross-domain SER in providing reliable emotion labels for TTS data when domain shift exists?
• RQ3 Does an auxiliary emotion prediction task based on style token weights improve the disentanglement of emotional content in the GST module?
• RQ4 Can the top-K reference selection strategy enhance emotional consistency and expressiveness compared to using all predicted audios?
• RQ5 To what extent does the proposed method preserve speech quality while enabling emotional control?

Key findings

• The proposed method achieves a mean opinion score (MOS) of 4.12 for neutral, 3.80 for angry, 3.11 for happy, and 3.61 for sad emotions, with no significant difference in quality compared to the baseline (p-value = 0.20).
• The average emotion recognition accuracy for four emotion categories (neutral, angry, happy, sad) reaches 78.75% with the proposed top-K scheme, significantly outperforming the baseline (36.75%) and the full-set baseline (49.25%).
• For arousal and valence dimensions, the average recognition accuracy is 91.0% and 55.5%, respectively, both well above the random baseline of 50.0%.
• t-SNE visualization confirms that style token weights for the proposed model form clear, distinct clusters per emotion category, unlike the baseline model which shows poor clustering.
• The model performs particularly poorly on 'happy' emotion, with a MOS of 3.11, likely due to lower SER accuracy for this class, highlighting the sensitivity of the method to SER quality.
• The auxiliary emotion prediction task significantly improves the GST module's ability to extract emotion-related features, as evidenced by improved clustering and recognition performance.