[Paper Review] Evaluation of Error and Correlation-Based Loss Functions For Multitask Learning Dimensional Speech Emotion Recognition

Bagus Tris Atmaja, Masato Akagi|arXiv (Cornell University)|Mar 24, 2020

Emotion and Mood Recognition16 references43 citations

TL;DR

This paper evaluates error-based (MSE, MAE) and correlation-based (CCC) loss functions for multitask learning in dimensional speech emotion recognition. Using two datasets (IEMOCAP, MSP-IMPROV) and two acoustic feature sets (GeMAPS, pAA), it demonstrates that CCC loss consistently outperforms MSE and MAE across all scenarios, achieving higher concordance correlation coefficient (CCC) scores on valence, arousal, and dominance dimensions. The key contribution is empirical validation that correlation-based loss is more suitable for regression-based dimensional emotion recognition than traditional error-based losses.

ABSTRACT

The choice of a loss function is a critical part of machine learning. This paper evaluated two different loss functions commonly used in regression-task dimensional speech emotion recognition, an error-based and a correlation-based loss functions. We found that using a correlation-based loss function with a concordance correlation coefficient (CCC) loss resulted in better performance than an error-based loss function with mean squared error (MSE) loss and mean absolute error (MAE), in terms of the averaged CCC score. The results are consistent with two input feature sets and two datasets. The scatter plots of test prediction by those two loss functions also confirmed the results measured by CCC scores.

Motivation & Objective

• To evaluate the impact of different loss functions—MSE, MAE, and CCC—on multitask learning in dimensional speech emotion recognition.
• To determine whether correlation-based loss (CCC) is more effective than error-based losses (MSE, MAE) for regression tasks in emotion recognition.
• To assess the consistency of performance across diverse datasets (IEMOCAP, MSP-IMPROV) and acoustic feature sets (GeMAPS, pAA).
• To validate the hypothesis that CCC loss, which accounts for both correlation and bias, leads to better model generalization than MSE and MAE.

Proposed method

• The study employs a multitask learning framework to jointly predict three emotional dimensions: valence, arousal, and dominance (VAD) using deep neural networks.
• Three loss functions are evaluated: mean squared error (MSE), mean absolute error (MAE), and concordance correlation coefficient loss (1 − CCC), with class-weights applied to balance valence and arousal.
• Two datasets—IEMOCAP and MSP-IMPROV—are used, with speaker-independent test splits (LOSO) and standardized label scaling from [1,5] to [-1,1].
• Two high-level statistical feature sets (HSF) are extracted: one from GeMAPS (23 LLDs) and one from pAA (34 LLDs), using only mean and standard deviation of low-level descriptors.
• Model performance is evaluated using averaged CCC across the three dimensions, with additional analysis of individual dimension scores and scatter plots for visual validation.
• The same neural architecture and hyperparameters are used across all experiments to ensure fair comparison between loss functions.

Experimental results

Research questions

• RQ1 Does using a correlation-based loss function (CCC) lead to better performance than error-based losses (MSE, MAE) in multitask dimensional speech emotion recognition?
• RQ2 Is the performance advantage of CCC loss consistent across different datasets and acoustic feature sets?
• RQ3 How do the prediction distributions of MSE, MAE, and CCC loss compare to the ground-truth labels in terms of alignment and bias?
• RQ4 Can CCC loss improve model generalization by minimizing both correlation and location bias in regression outputs?
• RQ5 Does the use of general-purpose feature sets (pAA) yield comparable performance to affective-specific feature sets (GeMAPS) when using CCC loss?

Key findings

• CCC loss achieved the highest average CCC score of 0.400 on IEMOCAP with GeMAPS features, significantly outperforming MSE (0.310) and MAE (0.304).
• On the MSP-IMPROV dataset with pAA features, CCC loss achieved an average CCC of 0.340, compared to 0.305 for MSE and 0.324 for MAE.
• The scatter plots of valence-arousal predictions confirmed that CCC loss predictions were more aligned with gold-standard labels than those from MSE and MAE.
• Performance consistency was observed across all four scenarios (2 datasets × 2 feature sets), with CCC loss consistently outperforming error-based losses in all cases.
• The averaged MSE scores were nearly identical across all configurations, suggesting that MSE may be less sensitive to model differences and less suitable for performance tracking across diverse setups.
• No significant performance gap was found between affective-designed GeMAPS and general-purpose pAA feature sets when using CCC loss, indicating robustness of the loss function across feature types.