[Paper Review] emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation

Ziyang Ma, Zhisheng Zheng+5 more|arXiv (Cornell University)|Dec 23, 2023

Emotion and Mood Recognition6 citations

TL;DR

emotion2vec is a self-supervised, universal speech emotion representation model pre-trained on 262 hours of unlabeled emotion data using online distillation with combined utterance-level and frame-level losses. It achieves state-of-the-art performance on IEMOCAP and generalizes across 10 languages and multiple emotion tasks, outperforming both general-purpose and emotion-specialized models with simple linear head fine-tuning.

ABSTRACT

We propose emotion2vec, a universal speech emotion representation model. emotion2vec is pre-trained on open-source unlabeled emotion data through self-supervised online distillation, combining utterance-level loss and frame-level loss during pre-training. emotion2vec outperforms state-of-the-art pre-trained universal models and emotion specialist models by only training linear layers for the speech emotion recognition task on the mainstream IEMOCAP dataset. In addition, emotion2vec shows consistent improvements among 10 different languages of speech emotion recognition datasets. emotion2vec also shows excellent results on other emotion tasks, such as song emotion recognition, emotion prediction in conversation, and sentiment analysis. Comparison experiments, ablation experiments, and visualization comprehensively demonstrate the universal capability of the proposed emotion2vec. To the best of our knowledge, emotion2vec is the first universal representation model in various emotion-related tasks, filling a gap in the field.

Motivation & Objective

• To address the lack of a universal speech emotion representation model capable of generalizing across diverse emotion tasks and languages.
• To improve speech emotion representation learning by combining utterance-level and frame-level self-supervised losses during pre-training.
• To develop a model that outperforms both general-purpose SSL models and task-specific emotion models without extensive fine-tuning.
• To demonstrate zero-shot transferability of learned representations to downstream tasks such as SER, song emotion recognition, and sentiment analysis.
• To validate the effectiveness of online distillation and multi-level supervision in learning emotion-aware representations.

Proposed method

• Pre-trains emotion2vec on 262 hours of open-source unlabeled emotion data using an online distillation paradigm to enable self-supervised learning.
• Integrates both utterance-level and frame-level contrastive losses during pre-training to capture global emotional context and local acoustic details.
• Uses chunk embeddings as the positive sample for utterance-level loss, which outperforms token-level and global-level methods.
• Applies a weighted combination of losses with a 1:1 ratio between utterance-level and frame-level loss, achieving optimal performance.
• Employs UMAP visualization to analyze learned representations and compare class separability across models.
• Trains a simple linear classifier on top of emotion2vec features for downstream evaluation, ensuring fair comparison with baselines.

Experimental results

Research questions

• RQ1 Can a self-supervised pre-training approach with joint utterance-level and frame-level losses learn more discriminative speech emotion representations than existing SSL models?
• RQ2 Does emotion2vec generalize across multiple languages and emotion tasks beyond speech emotion recognition?
• RQ3 How does the combination of utterance-level and frame-level supervision improve representation quality compared to single-supervision strategies?
• RQ4 Can emotion2vec achieve state-of-the-art performance on IEMOCAP with only a linear head fine-tuning, outperforming both general SSL models and emotion-specialist models?
• RQ5 How do the learned representations from emotion2vec compare visually and quantitatively in terms of intra-class compactness and inter-class margin?

Key findings

• emotion2vec achieves 72.13% weighted accuracy on the IEMOCAP dataset using only a linear head, surpassing all mainstream SSL models and specialist models.
• The model with both utterance-level and frame-level losses (71.79% WA) significantly outperforms models using only frame-level loss (70.85% WA) or only utterance-level loss (28.96% WA).
• emotion2vec shows consistent improvements across 13 diverse datasets spanning 10 languages, demonstrating strong language generalization capability.
• On song emotion recognition and emotion prediction in conversation, emotion2vec features achieve strong performance, confirming its task generalization ability.
• UMAP visualizations show that emotion2vec features exhibit higher intra-class compactness and larger inter-class margins than WavLM and data2vec, indicating better class discriminability.
• Ablation studies confirm that the 1:1 loss weighting and chunk embedding-based utterance-level loss yield the best performance, validating the design choices.