[Paper Review] Multimodal Fusion with Deep Neural Networks for Audio-Video Emotion Recognition

Juan Diego Ortega, Mohammed Senoussaoui+4 more|arXiv (Cornell University)|Jul 6, 2019

Speech and Audio ProcessingComputer Science27 references43 citations

TL;DR

A novel deep neural network for end-to-end multimodal fusion of audio, video, and text for continuous emotion prediction, achieving higher CCC than early/late fusion baselines on AVEC SEWA data.

ABSTRACT

This paper presents a novel deep neural network (DNN) for multimodal fusion of audio, video and text modalities for emotion recognition. The proposed DNN architecture has independent and shared layers which aim to learn the representation for each modality, as well as the best combined representation to achieve the best prediction. Experimental results on the AVEC Sentiment Analysis in the Wild dataset indicate that the proposed DNN can achieve a higher level of Concordance Correlation Coefficient (CCC) than other state-of-the-art systems that perform early fusion of modalities at feature-level (i.e., concatenation) and late fusion at score-level (i.e., weighted average) fusion. The proposed DNN has achieved CCCs of 0.606, 0.534, and 0.170 on the development partition of the dataset for predicting arousal, valence and liking, respectively.

Motivation & Objective

• Motivate robust emotion recognition in the wild using multiple modalities (audio, video, text).
• Design a DNN that learns modality-specific representations and a fused representation jointly.
• Evaluate the proposed architecture against early fusion (feature-level) and late fusion (score-level) baselines on AVEC SEWA/RECOLA datasets.

Proposed method

• Proposes a DNN with independent modality-specific layers to learn discriminant representations.
• Uses a concatenation-based merge layer to fuse modality representations, followed by fully connected and regression layers.
• Employs a single linear neuron for regression and a scaling module to align prediction and label magnitudes.
• Trains with Mean Squared Error loss and uses Keras checkpointing to select the best model.
• Explores several post-processing scalings (min-max, standard deviation ratio, decimal scaling) to improve CCC.
• Evaluates on arousal, valence, and liking dimensions with separate architectures per dimension.

Experimental results

Research questions

• RQ1 Can a hybrid DNN architecture learn effective modality-specific representations and a joint fusion for continuous emotion prediction?
• RQ2 Does end-to-end multimodal fusion outperform traditional early (feature-level) and late (score-level) fusion in realistic conditions?
• RQ3 What is the impact of preprocessing and postprocessing steps (delay compensation, scaling) on CCC for arousal, valence, and liking?
• RQ4 How does the proposed architecture perform across audio, video, and text modalities individually and in fusion?
• RQ5 Is temporal modeling (e.g., recurrent components) necessary to further improve accuracy beyond the proposed feedforward fusion?

Key findings

• The proposed DNN achieves higher Concordance Correlation Coefficient (CCC) than state-of-the-art early and late fusion baselines on AVEC SEWA.
• Development-set CCCs reached up to 0.606 (arousal and liking), and 0.534 (valence) for the proposed fusion architecture.
• Text modality is highly informative, especially for liking, and fusion benefits from combining modalities rather than relying on a single source.
• Delay compensation and postprocessing scaling significantly affect CCC, with optimized delay (1.5s for arousal/valence, 2.5s for liking) improving performance.
• The architecture enables end-to-end optimization of features, classifiers, and fusion function, outperforming unimodal and early/late fusion baselines on development data.
• Test-set results indicate some drop possibly due to overfitting, suggesting need for further temporal modeling or regularization.