[Paper Review] Audio-video Emotion Recognition in the Wild using Deep Hybrid Networks

Xin Guo, Luisa F. Polanía+1 more|arXiv (Cornell University)|Feb 20, 2020

Speech and Audio Processing25 references8 citations

TL;DR

This paper proposes a deep hybrid neural network for audio-visual emotion recognition in the wild, combining VGG-LSTM models on facial images, SVM and LSTM on audio features, and an Inception(v2)-LSTM on audio-derived image maps. The method achieves 55.61% accuracy on the validation set and 51.15% on the test set, significantly outperforming the EmotiW 2017 baseline of 38.81%.

ABSTRACT

This paper presents an audiovisual-based emotion recognition hybrid network. While most of the previous work focuses either on using deep models or hand-engineered features extracted from images, we explore multiple deep models built on both images and audio signals. Specifically, in addition to convolutional neural networks (CNN) and recurrent neutral networks (RNN) trained on facial images, the hybrid network also contains one SVM classifier trained on holistic acoustic feature vectors, one long short-term memory network (LSTM) trained on short-term feature sequences extracted from segmented audio clips, and one Inception(v2)-LSTM network trained on image-like maps, which are built based on short-term acoustic feature sequences. Experimental results show that the proposed hybrid network outperforms the baseline method by a large margin.

Motivation & Objective

• Address the challenge of emotion recognition in unconstrained, real-world video settings with noisy, variable, and uncontrolled conditions.
• Overcome limitations of prior methods that focus primarily on visual features, by fully exploiting audio channel information.
• Improve performance on the EmotiW 2017 audio-video emotion recognition sub-challenge by integrating multimodal deep learning models with late fusion.
• Demonstrate that combining audio and visual models, especially through sequence modeling and feature map representation, enhances recognition accuracy.

Proposed method

• Train a VGG-16-based CNN-LSTM network on aligned facial images extracted at 8-frame intervals to model temporal dynamics of facial expressions.
• Apply a two-stream VGG-LSTM architecture: one trained on faces detected via a custom method, and another on faces provided by the challenge organizers, to improve robustness.
• Train an SVM classifier on holistic acoustic feature vectors extracted via openSmile for audio-based emotion recognition.
• Implement an LSTM network on short-term audio feature sequences to model temporal dynamics in audio modality.
• Construct image-like maps from stacked short-term acoustic features and train an Inception(v2)-LSTM network to learn spatiotemporal patterns in audio representations.
• Fuse predictions from all five models (two VGG-LSTM, one SVM, one LSTM, one Inception(v2)-LSTM) using grid search-optimized decision-level fusion to maximize validation accuracy.

Experimental results

Research questions

• RQ1 Can a hybrid deep learning architecture that combines visual and audio modalities significantly improve emotion recognition accuracy in unconstrained video settings?
• RQ2 How does the use of image-like maps derived from short-term audio features compare to raw audio sequences in modeling emotional dynamics?
• RQ3 Does end-to-end training of CNN-LSTM on facial images with frame gaps outperform feature extraction followed by separate RNN training?
• RQ4 To what extent do audio-based models, particularly those using LSTM and SVM on acoustic features, contribute to overall performance when fused with visual models?
• RQ5 Can late fusion of multiple deep models trained on different representations of audio and video lead to superior performance on the EmotiW 2017 benchmark?

Key findings

• The proposed hybrid network achieves a validation set accuracy of 55.61%, representing a significant improvement over the EmotiW 2017 audio-video emotion recognition sub-challenge baseline of 38.81%.
• The best-performing VGG-LSTM model on aligned faces reaches 49.61% accuracy, outperforming the 2016 sub-challenge winner’s 45.43% accuracy.
• The combination of two VGG-LSTM models (trained on different face detection methods) achieves 51.02% accuracy on the validation set, demonstrating the benefit of multi-source face detection.
• Audio-based models, particularly the Inception(v2)-LSTM on audio maps, achieve 26.63% validation accuracy and show strong performance on the anger class, contributing to overall fusion gains.
• The final hybrid network achieves 51.15% accuracy on the test set, with a weighted average per-class accuracy of 41.21%, indicating robust generalization across emotion classes.
• The confusion matrix on the test set shows that the model performs best on happiness (63.89% recall) and neutral (50.78% recall), while struggling most with surprise (0% recall), suggesting class-specific challenges.