[Paper Review] Mutilmodal Feature Extraction and Attention-based Fusion for Emotion Estimation in Videos

Shu Tao, Xinke Wang+4 more|arXiv (Cornell University)|Mar 18, 2023

Emotion and Mood Recognition4 citations

TL;DR

This paper proposes an attention-based multimodal fusion framework for video-based emotion estimation, integrating visual, audio, and pose features from videos of varying lengths. By applying cross-attention mechanisms to temporal features and concatenating attended representations, the model achieves a validation F1 score of 0.361 on the ABAW 2023 benchmark, outperforming early fusion and single-modality baselines.

ABSTRACT

The continuous improvement of human-computer interaction technology makes it possible to compute emotions. In this paper, we introduce our submission to the CVPR 2023 Competition on Affective Behavior Analysis in-the-wild (ABAW). Sentiment analysis in human-computer interaction should, as far as possible Start with multiple dimensions, fill in the single imperfect emotion channel, and finally determine the emotion tendency by fitting multiple results. Therefore, We exploited multimodal features extracted from video of different lengths from the competition dataset, including audio, pose and images. Well-informed emotion representations drive us to propose a Attention-based multimodal framework for emotion estimation. Our system achieves the performance of 0.361 on the validation dataset. The code is available at [https://github.com/xkwangcn/ABAW-5th-RT-IAI].

Motivation & Objective

• To improve affective behavior analysis in the wild by leveraging multiple modalities—visual, audio, and pose—for more robust emotion estimation.
• To address the limitations of single-modality approaches by fusing complementary information from diverse data streams.
• To enhance temporal modeling of emotion dynamics through attention mechanisms that weigh relevant contextual features across modalities.
• To achieve state-of-the-art performance on the ABAW 2023 Affective Behavior Analysis in-the-wild challenge.

Proposed method

• Extracted facial features using a pre-trained FAN (Face Alignment Network) on AffectNet, then fine-tuned on Aff-Wild2 for improved facial representation learning.
• Extracted audio features using DenseNet121 with a 1-second window and 500ms hop size, producing 1024-dimensional embeddings.
• Extracted 2D pose coordinates using OpenPose at 0.5s intervals to generate dynamic pose sequences.
• Applied two-layer RNNs to model temporal dependencies in facial, audio, and pose feature sequences independently.
• Implemented cross-attention modules that attend to temporal context from each modality using the current frame’s facial feature as a query.
• Concatenated the three attended feature representations with the current facial feature and passed them through a fully connected layer for expression prediction.

Experimental results

Research questions

• RQ1 Can attention-based fusion of multimodal features (visual, audio, pose) improve emotion estimation performance compared to early or late fusion?
• RQ2 How does the integration of long-range temporal context from multiple modalities enhance the robustness of emotion recognition in unconstrained video settings?
• RQ3 What is the impact of using cross-attention mechanisms to dynamically weigh relevant contextual information from other modalities during inference?
• RQ4 Does the proposed framework generalize across different data splits and maintain high performance on the ABAW 2023 validation set?

Key findings

• The attention-based fusion model achieved a validation F1 score of 0.361, outperforming both early fusion (0.318) and single-modality baselines.
• The model achieved the highest average F1 score on the 5th split (35.5%) and the best overall performance on the official split (33.7%).
• The attention fusion strategy significantly improved performance on challenging classes such as 'Anger' (32% F1) and 'Surprise' (28% F1), which are often under-predicted.
• The use of cross-attention allowed the model to dynamically attend to relevant temporal context from audio, pose, and visual sequences, improving feature representation quality.
• The model demonstrated strong generalization across five random data splits, with F1 scores ranging from 30.0% to 35.5%.
• Fine-tuning the FAN on Aff-Wild2 improved facial feature quality, contributing to the overall performance gain.