[Paper Review] Deep Multilayer Perceptrons for Dimensional Speech Emotion Recognition

Bagus Tris Atmaja, Masato Akagi|arXiv (Cornell University)|Apr 6, 2020

Emotion and Mood RecognitionPsychology17 references17 citations

TL;DR

This paper proposes a deep multilayer perceptron (MLP) with reduced input dimensionality using high-level statistical functions (HSF) to achieve state-of-the-art performance in dimensional speech emotion recognition, outperforming deeper architectures like LSTM and CNN on IEMOCAP and MSP-IMPROV datasets across both speaker-dependent and speaker-independent settings, despite requiring significantly less computational power.

ABSTRACT

Modern deep learning architectures are ordinarily performed on high-performance computing facilities due to the large size of the input features and complexity of its model. This paper proposes traditional multilayer perceptrons (MLP) with deep layers and small input size to tackle that computation requirement limitation. The result shows that our proposed deep MLP outperformed modern deep learning architectures, i.e., LSTM and CNN, on the same number of layers and value of parameters. The deep MLP exhibited the highest performance on both speaker-dependent and speaker-independent scenarios on IEMOCAP and MSP-IMPROV corpus.

Motivation & Objective

• To address the high computational demands of modern deep learning in speech emotion recognition by proposing a lightweight deep MLP architecture.
• To evaluate whether a traditional deep MLP can surpass modern architectures like LSTM and CNN when using compact, high-level statistical features.
• To investigate the effectiveness of deep MLPs in both speaker-dependent and speaker-independent speech emotion recognition scenarios.
• To demonstrate that high performance in dimensional emotion recognition does not require expensive GPU-based hardware or complex models.
• To generalize findings across multiple datasets, including within-corpus and mixed-corpus evaluations.

Proposed method

• The input features are derived from the GeMAPS low-level descriptors (LLDs), with high-level statistical functions (HSF) applied per utterance to reduce dimensionality from 3409×23 to 1×23.
• A deep multilayer perceptron (MLP) with five hidden layers is used, excluding input and output layers, trained using mean squared error (MSE) loss.
• The model predicts three continuous emotion dimensions: valence, arousal, and dominance, normalized to [-1, 1] for training.
• Two training scenarios are evaluated: speaker-dependent (80/20 split) and speaker-independent (leave-one-session-out, LOSO), using IEMOCAP and MSP-IMPROV datasets.
• The model is evaluated using the concordance correlation coefficient (CCC) as the primary metric across all settings.
• A mixed-corpus dataset is created by concatenating training, development, and test sets from IEMOCAP and MSP-IMPROV to assess cross-dataset generalization.

Experimental results

Research questions

• RQ1 Can a deep MLP with small input size outperform modern deep learning architectures like LSTM and CNN in dimensional speech emotion recognition?
• RQ2 How does the performance of the proposed deep MLP compare between speaker-dependent and speaker-independent settings across multiple datasets?
• RQ3 To what extent does the use of high-level statistical features (HSF) reduce computational complexity while preserving recognition accuracy?
• RQ4 How does the model generalize across different datasets when combined into a mixed-corpus evaluation?
• RQ5 Does the model's performance improve if trained with a loss function matching the evaluation metric (CCC) rather than MSE?

Key findings

• The proposed deep MLP achieved the highest concordance correlation coefficient (CCC) across all emotion dimensions and averaged scores in both IEMOCAP and MSP-IMPROV datasets under both speaker-dependent and speaker-independent settings.
• On IEMOCAP, the deep MLP achieved a mean CCC of 0.469 (speaker-dependent) and 0.453 (speaker-independent), outperforming LSTM (0.387 and 0.359) and CNN (0.307 and 0.328).
• On MSP-IMPROV, the deep MLP achieved a mean CCC of 0.536 (speaker-dependent) and 0.407 (speaker-independent), surpassing LSTM (0.515 and 0.335) and CNN (0.497 and 0.375).
• In the mixed-corpus evaluation, the deep MLP achieved a mean CCC of 0.499 (speaker-dependent) and 0.294 (speaker-independent), again outperforming LSTM and CNN across all dimensions.
• The performance gap between speaker-dependent and speaker-independent scenarios was smaller in IEMOCAP than in MSP-IMPROV, likely due to differences in file naming and session ordering.
• The model's superior performance, despite using MSE loss instead of CCC loss, suggests that further gains could be achieved with CCC-based training, indicating strong potential for optimization.