[Paper Review] Self-Supervised Learning for Audio-Based Emotion Recognition

Peranut Nimitsurachat, Peter Washington|arXiv (Cornell University)|Jul 23, 2023

Music and Audio ProcessingComputer Science3 citations

TL;DR

This paper proposes a self-supervised learning (SSL) approach for audio-based emotion recognition using encoded acoustic features (74-dimensional COVAREP embeddings) from the CMU-MOSEI dataset. By pre-training on unannotated data via masked timestamp prediction and fine-tuning on small labeled sets, the method significantly improves emotion classification performance—especially for easily identifiable emotions like happy, sad, and anger—demonstrating that SSL is most effective when labeled data is scarce.

ABSTRACT

Emotion recognition models using audio input data can enable the development of interactive systems with applications in mental healthcare, marketing, gaming, and social media analysis. While the field of affective computing using audio data is rich, a major barrier to achieve consistently high-performance models is the paucity of available training labels. Self-supervised learning (SSL) is a family of methods which can learn despite a scarcity of supervised labels by predicting properties of the data itself. To understand the utility of self-supervised learning for audio-based emotion recognition, we have applied self-supervised learning pre-training to the classification of emotions from the CMU- MOSEI's acoustic modality. Unlike prior papers that have experimented with raw acoustic data, our technique has been applied to encoded acoustic data. Our model is first pretrained to uncover the randomly-masked timestamps of the acoustic data. The pre-trained model is then fine-tuned using a small sample of annotated data. The performance of the final model is then evaluated via several evaluation metrics against a baseline deep learning model with an identical backbone architecture. We find that self-supervised learning consistently improves the performance of the model across all metrics. This work shows the utility of self-supervised learning for affective computing, demonstrating that self-supervised learning is most useful when the number of training examples is small, and that the effect is most pronounced for emotions which are easier to classify such as happy, sad and anger. This work further demonstrates that self-supervised learning works when applied to embedded feature representations rather than the traditional approach of pre-training on the raw input space.

Motivation & Objective

• Address the challenge of limited annotated data in audio-based emotion recognition, a major bottleneck in affective computing.
• Investigate whether self-supervised learning (SSL) can improve performance when training data is scarce, particularly in the audio modality.
• Explore the effectiveness of SSL not on raw audio, but on pre-computed, high-dimensional acoustic embeddings (74D COVAREP features).
• Quantify the performance gains of SSL across different emotion types and varying amounts of labeled data.
• Compare the pre-trained SSL model against a standard baseline with identical architecture to isolate the impact of pre-training.

Proposed method

• Pre-train a deep learning model on unannotated acoustic data by masking random timestamps in the 74-dimensional feature sequences and training the model to reconstruct them.
• Use a transformer-based backbone architecture for both pre-training and fine-tuning, ensuring architectural consistency with the baseline.
• Fine-tune the pre-trained model on small, labeled subsets of the CMU-MOSEI dataset using standard cross-entropy or regression loss for emotion intensity prediction.
• Evaluate performance using multiple metrics: overall and per-emotion mean absolute error (MAE), 4-class accuracy, and per-emotion accuracy.
• Train and evaluate models across multiple runs (three iterations per label size) to assess stability and generalization of SSL gains.
• Compare the pre-trained model’s performance against a non-pretrained baseline with identical architecture to isolate the effect of self-supervised pre-training.

Figure 1: Distribution of emotions in CMU-MOSEI

Experimental results

Research questions

• RQ1 Does self-supervised pre-training on encoded acoustic features improve emotion recognition performance compared to a standard supervised baseline?
• RQ2 How does the performance gain from SSL vary with the amount of labeled data available during fine-tuning?
• RQ3 Are the benefits of SSL more pronounced for certain emotion categories, such as 'easier' emotions (e.g., happy, sad, anger) versus more nuanced ones (e.g., surprise, fear)?
• RQ4 Can self-supervised learning effectively leverage unannotated data when applied to embedded feature representations rather than raw audio waveforms?
• RQ5 How does the convergence of the pre-trained model to the baseline model occur as the number of labeled examples increases?

Key findings

• Self-supervised learning consistently improves performance across all evaluation metrics—overall MAE, per-emotion MAE, 4-class accuracy, and per-emotion accuracy—especially when labeled data is scarce.
• The performance gain from SSL is most significant when only 0–200 labeled examples are available during fine-tuning, with gains diminishing as the number of labels increases.
• The model shows the most substantial improvements for 'easier' emotions such as happy, sad, and anger, while gains for more nuanced emotions like surprise and fear are smaller.
• The mean absolute error (MAE) for easier emotions drops significantly with SSL pre-training, while the MAE for nuanced emotions remains relatively low (0.12–0.16) even without pre-training, suggesting a performance ceiling.
• The pre-trained model converges toward the performance of the baseline model as the number of labeled examples increases, indicating that SSL provides a strong inductive bias that is most valuable in low-data regimes.
• Applying SSL to encoded acoustic features (74D COVAREP embeddings) rather than raw audio is effective and demonstrates that the method generalizes beyond raw input space, opening new avenues for feature-level pre-training in affective computing.

Figure 2: An overview of our proposed methodology for pre-training the deep learning model before transferring pre-trained model to the fine-tuning step