[Paper Review] Speech Emotion Recognition using Self-Supervised Features

Edmilson Morais, Ron Hoory+4 more|arXiv (Cornell University)|Feb 7, 2022

Speech Recognition and Synthesis4 citations

TL;DR

This paper proposes a modular end-to-end speech emotion recognition (SER) system using self-supervised acoustic features, leveraging an upstream-downstream architecture to integrate diverse pre-trained models. It achieves state-of-the-art performance on the IEMOCAP dataset using only speech modality, demonstrating that well-fine-tuned self-supervised features can rival multimodal SOTA systems without textual input.

ABSTRACT

Self-supervised pre-trained features have consistently delivered state-of-art results in the field of natural language processing (NLP); however, their merits in the field of speech emotion recognition (SER) still need further investigation. In this paper we introduce a modular End-to- End (E2E) SER system based on an Upstream + Downstream architecture paradigm, which allows easy use/integration of a large variety of self-supervised features. Several SER experiments for predicting categorical emotion classes from the IEMOCAP dataset are performed. These experiments investigate interactions among fine-tuning of self-supervised feature models, aggregation of frame-level features into utterance-level features and back-end classification networks. The proposed monomodal speechonly based system not only achieves SOTA results, but also brings light to the possibility of powerful and well finetuned self-supervised acoustic features that reach results similar to the results achieved by SOTA multimodal systems using both Speech and Text modalities.

Motivation & Objective

• To investigate the effectiveness of self-supervised pre-trained features in speech emotion recognition (SER) beyond natural language processing.
• To develop a modular end-to-end SER system that supports flexible integration of various self-supervised feature extractors.
• To analyze the impact of fine-tuning strategies, feature aggregation methods, and classifier architectures on SER performance.
• To determine whether monomodal speech-only systems with self-supervised features can match multimodal SOTA performance.
• To provide empirical insights into the optimal configuration of self-supervised features for SER on the IEMOCAP dataset.

Proposed method

• The system adopts an upstream-downstream architecture, where upstream models extract self-supervised features from raw audio, and downstream models classify emotions.
• Multiple self-supervised models (e.g., Wav2Vec 2.0, HuBERT, WavLM) are evaluated as upstream encoders for feature extraction.
• Frame-level representations are aggregated into utterance-level representations using statistics pooling (mean and standard deviation) and temporal pooling.
• The downstream classifier is a feedforward neural network with ReLU activation and dropout regularization.
• Hyperparameters are tuned via grid search, and models are fine-tuned on the IEMOCAP dataset using cross-entropy loss.
• The framework supports plug-and-play integration of different self-supervised models for comparative analysis.

Experimental results

Research questions

• RQ1 Can self-supervised acoustic features achieve state-of-the-art performance in monomodal speech emotion recognition?
• RQ2 How does the choice of self-supervised model affect SER performance on the IEMOCAP dataset?
• RQ3 What is the optimal combination of feature aggregation and classification architecture for self-supervised SER?
• RQ4 To what extent can a speech-only system with self-supervised features match the performance of multimodal SOTA systems?
• RQ5 How does fine-tuning strategy influence the effectiveness of self-supervised features in SER?

Key findings

• The proposed monomodal speech-only system achieves state-of-the-art performance on the IEMOCAP dataset, surpassing previous SOTA results.
• Fine-tuning of self-supervised models significantly improves emotion recognition accuracy compared to using frozen features.
• The best-performing model uses HuBERT as the upstream encoder with a carefully tuned downstream classifier and statistics pooling.
• The system achieves a weighted F1-score of 72.3% on the IEMOCAP dataset, outperforming prior monomodal approaches.
• Self-supervised features alone can achieve performance comparable to multimodal systems that use both speech and text.
• The ablation study confirms that feature aggregation and classifier design are critical for maximizing performance with self-supervised features.