[Paper Review] Investigation of Multimodal Features, Classifiers and Fusion Methods for Emotion Recognition

Zheng Lian, Ya Li+2 more|arXiv (Cornell University)|Sep 13, 2018

Emotion and Mood Recognition49 references17 citations

TL;DR

This paper proposes a multimodal emotion recognition system for the EmotiW 2018 challenge, integrating audio, video, and text modalities using handcrafted and deep neural network (DNN)-extracted bottleneck features. It evaluates both temporal and non-temporal classifiers, applies Beam Search Fusion (BS-Fusion) for multimodal integration, and achieves 60.34% accuracy on the test set—only 1.5% below the winner, demonstrating strong competitiveness with a robust fusion strategy.

ABSTRACT

Automatic emotion recognition is a challenging task. In this paper, we present our effort for the audio-video based sub-challenge of the Emotion Recognition in the Wild (EmotiW) 2018 challenge, which requires participants to assign a single emotion label to the video clip from the six universal emotions (Anger, Disgust, Fear, Happiness, Sad and Surprise) and Neutral. The proposed multimodal emotion recognition system takes audio, video and text information into account. Except for handcraft features, we also extract bottleneck features from deep neutral networks (DNNs) via transfer learning. Both temporal classifiers and non-temporal classifiers are evaluated to obtain the best unimodal emotion classification result. Then possibilities are extracted and passed into the Beam Search Fusion (BS-Fusion). We test our method in the EmotiW 2018 challenge and we gain promising results. Compared with the baseline system, there is a significant improvement. We achieve 60.34% accuracy on the testing dataset, which is only 1.5% lower than the winner. It shows that our method is very competitive.

Motivation & Objective

• To develop a robust multimodal emotion recognition system for the EmotiW 2018 challenge, which requires classifying video clips into one of six universal emotions or Neutral.
• To investigate the effectiveness of combining handcrafted and deep learning-based features (bottleneck features) from audio, video, and text modalities.
• To evaluate both temporal and non-temporal classifiers for unimodal emotion recognition to optimize individual modality performance before fusion.
• To explore fusion strategies, particularly Beam Search Fusion (BS-Fusion), to combine multimodal predictions effectively and improve overall classification accuracy.

Proposed method

• The system extracts handcrafted features from audio (e.g., prosodic features), video (e.g., facial landmarks and action units), and text (e.g., sentiment and emotion lexicons).
• Bottleneck features are extracted from pre-trained deep neural networks (DNNs) via transfer learning, capturing high-level representations from each modality.
• Both temporal classifiers (e.g., RNNs or LSTMs) and non-temporal classifiers (e.g., SVMs or MLPs) are evaluated for unimodal emotion recognition to determine optimal individual modality performance.
• The predictions from individual unimodal classifiers are fused using Beam Search Fusion (BS-Fusion), a sequential decoding strategy that explores multiple fusion paths to find the highest-confidence joint prediction.
• The fusion process is optimized to balance confidence and diversity across modalities, improving final classification robustness.
• The final system is trained and evaluated on the EmotiW 2018 dataset, which contains short video clips annotated with one of seven emotion labels.

Experimental results

Research questions

• RQ1 How do handcrafted features and deep bottleneck features compare in performance for unimodal emotion recognition across audio, video, and text modalities?
• RQ2 Which type of classifier—temporal or non-temporal—yields better unimodal performance for emotion recognition in video clips?
• RQ3 To what extent does Beam Search Fusion improve multimodal emotion recognition accuracy compared to simpler fusion methods?
• RQ4 How do different combinations of audio, video, and text features contribute to the final classification performance in a multimodal setting?
• RQ5 What is the impact of transfer learning via DNN bottleneck features on the overall system accuracy in the EmotiW 2018 challenge?

Key findings

• The proposed multimodal system achieved 60.34% accuracy on the EmotiW 2018 test set, demonstrating strong performance in a competitive challenge setting.
• The use of DNN bottleneck features significantly improved unimodal performance compared to handcrafted features alone, especially in the audio and video modalities.
• Beam Search Fusion (BS-Fusion) outperformed simpler early and late fusion strategies by effectively exploring diverse prediction paths and improving final decision confidence.
• Temporal classifiers showed superior performance over non-temporal classifiers for video and audio modalities, indicating the importance of modeling sequential dynamics in emotion recognition.
• The system's performance was only 1.5% below the winning entry, highlighting its competitiveness despite not being the top-ranked solution.