[Paper Review] Multimodal Speech Emotion Recognition and Ambiguity Resolution

Gaurav Sahu|arXiv (Cornell University)|Apr 12, 2019

Music and Audio Processing19 references42 citations

TL;DR

The paper compares lightweight hand-crafted feature-based ML models with deep learning for multimodal (audio and text) speech emotion recognition on IEMOCAP, showing that simple ML ensembles can rival DL methods and that adding text improves performance.

ABSTRACT

Identifying emotion from speech is a non-trivial task pertaining to the ambiguous definition of emotion itself. In this work, we adopt a feature-engineering based approach to tackle the task of speech emotion recognition. Formalizing our problem as a multi-class classification problem, we compare the performance of two categories of models. For both, we extract eight hand-crafted features from the audio signal. In the first approach, the extracted features are used to train six traditional machine learning classifiers, whereas the second approach is based on deep learning wherein a baseline feed-forward neural network and an LSTM-based classifier are trained over the same features. In order to resolve ambiguity in communication, we also include features from the text domain. We report accuracy, f-score, precision, and recall for the different experiment settings we evaluated our models in. Overall, we show that lighter machine learning based models trained over a few hand-crafted features are able to achieve performance comparable to the current deep learning based state-of-the-art method for emotion recognition.

Motivation & Objective

• Motivate the study of emotion recognition despite ambiguous emotion definitions.
• Compare traditional ML models trained on hand-crafted audio features with DL models trained on the same features.
• Investigate the impact of text modality and audio+text fusion on emotion recognition accuracy.
• Evaluate models under audio-only, text-only, and multimodal settings on IEMOCAP.
• Provide insights into feature importance for multimodal SER and discuss future improvements.

Proposed method

• Extract eight hand-crafted time-domain audio features (pitch, harmonics, energy, pause, central moments).
• Compute TFIDF features from text transcripts.
• Train traditional ML classifiers (Random Forest, Gradient Boosting, SVM, Naive Bayes, Logistic Regression) and compare with DL models (MLP, LSTM) using the same audio features.
• Implement simple ensemble methods combining RF, XGBoost, and MLP (and extended ensemble with MNB and LR).
• Fuse audio and text features by simple concatenation for the multimodal setting.
• Evaluate using accuracy, precision, recall, and F1-score across three settings: Audio-only, Text-only, Audio+Text.

Experimental results

Research questions

• RQ1 How do hand-crafted audio features with traditional ML compare to DL models on IEMOCAP for six-class emotion recognition?
• RQ2 Does incorporating textual data improve SER performance and how does multimodal fusion affect results?
• RQ3 Which features contribute most to emotion prediction, and does fusion help resolve modality-specific ambiguities?
• RQ4 What is the impact of modality (audio, text, multimodal) on confusion among emotion classes?
• RQ5 Can lightweight models achieve performance close to or exceeding state-of-the-art DL approaches on this task?

Key findings

• Lightweight ML ensembles trained on eight audio features achieve comparable performance to deeper DL models on IEMOCAP.
• Audio-only results show LSTM underperforming relative to the ensemble E1 in several settings, particularly for distinguishing neutral and some closely related emotions.
• Text-only models perform well, with TRE (text encoder) performing robustly across six emotion classes.
• Audio+Text fusion yields about a 14% improvement in metrics over single-modality models, indicating strong inter-modal information coupling.
• Text features help correctly classify angry and happy, while audio features improve sadness detection.
• Harmonics and pause emerge as the most important audio features for prediction among the eight features used.