[Paper Review] Textless Speech Emotion Conversion using Discrete and Decomposed Representations

Felix Kreuk, Adam Polyak+8 more|arXiv (Cornell University)|Nov 14, 2021

Speech and Audio Processing4 citations

TL;DR

This paper proposes a textless speech emotion conversion method that treats emotion conversion as a spoken language translation task using discrete, decomposed speech representations. By disentangling content, prosody, speaker, and emotion, the model translates phonetic units to target emotions and predicts prosodic features, achieving state-of-the-art performance in perceived emotion and audio quality, even surpassing text-based systems in subjective evaluations.

ABSTRACT

Speech emotion conversion is the task of modifying the perceived emotion of a speech utterance while preserving the lexical content and speaker identity. In this study, we cast the problem of emotion conversion as a spoken language translation task. We use a decomposition of the speech signal into discrete learned representations, consisting of phonetic-content units, prosodic features, speaker, and emotion. First, we modify the speech content by translating the phonetic-content units to a target emotion, and then predict the prosodic features based on these units. Finally, the speech waveform is generated by feeding the predicted representations into a neural vocoder. Such a paradigm allows us to go beyond spectral and parametric changes of the signal, and model non-verbal vocalizations, such as laughter insertion, yawning removal, etc. We demonstrate objectively and subjectively that the proposed method is vastly superior to current approaches and even beats text-based systems in terms of perceived emotion and audio quality. We rigorously evaluate all components of such a complex system and conclude with an extensive model analysis and ablation study to better emphasize the architectural choices, strengths and weaknesses of the proposed method. Samples are available under the following link: https://speechbot.github.io/emotion.

Motivation & Objective

• To address the challenge of generating expressive, emotionally rich speech while preserving lexical content and speaker identity.
• To overcome limitations of signal-based methods that cannot model non-verbal vocalizations and text-based methods that lack annotations for such cues.
• To enable high-fidelity emotion conversion without requiring text transcriptions, by leveraging disentangled speech representations.
• To demonstrate that a textless, representation-based approach can match or exceed text-based systems in perceived emotion and audio quality.
• To provide a comprehensive ablation and analysis of component contributions to the full system.

Proposed method

• The system uses a speech tokenizer to extract discrete content units (e.g., phonetic-content) from raw speech, enabling discrete sequence modeling.
• A sequence-to-sequence (S2S) model translates content units from a source emotion to a target emotion, treating emotion conversion as a spoken language translation task.
• Prosodic features—pitch (F0) and duration—are predicted based on the translated content units and target emotion embedding.
• The final speech waveform is synthesized using a neural vocoder conditioned on the translated content, predicted prosody, speaker identity, and target emotion.
• The method decomposes speech into four disentangled components: content, prosody, speaker, and emotion, enabling independent manipulation.
• The entire pipeline is trained in a parallel data setting, with end-to-end optimization across all components.

Experimental results

Research questions

• RQ1 Can a textless approach achieve superior emotion conversion performance by modeling non-verbal vocalizations such as laughter and yawning?
• RQ2 How does disentangling speech representations into content, prosody, speaker, and emotion improve emotion conversion quality?
• RQ3 To what extent does the proposed method outperform both signal-based and text-based emotion conversion baselines in perceived emotion and audio quality?
• RQ4 Which components of the system (content translation, prosody prediction, emotion conditioning) are most critical for successful emotion transfer?
• RQ5 How generalizable is the model to out-of-distribution utterances, such as neutral speech from LibriSpeech, when converted to expressive emotions?

Key findings

• The proposed method achieved an eMOC score of 86.11% for converting to angry and 88.01% for neutral, outperforming baseline models.
• For out-of-distribution LibriSpeech samples, the model achieved an average eMOC score of 82.25% and MOS of 3.69, indicating strong generalization.
• The system reached 79.12% eMOC for Amused and 86.11% for Angry, showing high success in emotion transfer, with performance close to ground-truth recordings.
• The ablation study showed that predicting F0 and durations alone achieved 70.31% eMOC for Sleepy, while full pipeline performance matched ground-truth levels.
• The model significantly outperformed text-based systems in perceived emotion and audio quality, with MOS and eMOC scores exceeding those of text-based baselines.
• The system successfully modeled non-verbal vocalizations such as laughter insertion and yawning removal, which are typically missed by signal- and text-based approaches.