[Paper Review] Spatial and Temporal Networks for Facial Expression Recognition in the Wild Videos

Shuyi Mao, Xinqi Fan+1 more|arXiv (Cornell University)|Jul 12, 2021

Emotion and Mood Recognition16 references4 citations

TL;DR

This paper proposes an ensemble model combining ResNet50, CNN-RNN, and CNN-Transformer architectures to improve facial expression recognition in the wild using spatial and temporal modeling. Pretraining on VGGFace2 enhances robustness to in-the-wild variations, and the ensemble achieves an F1-score of 0.4133, accuracy of 0.6216, and final metric of 0.4821 on the ABAW 2021 validation set.

ABSTRACT

The paper describes our proposed methodology for the seven basic expression classification track of Affective Behavior Analysis in-the-wild (ABAW) Competition 2021. In this task, facial expression recognition (FER) methods aim to classify the correct expression category from a diverse background, but there are several challenges. First, to adapt the model to in-the-wild scenarios, we use the knowledge from pre-trained large-scale face recognition data. Second, we propose an ensemble model with a convolution neural network (CNN), a CNN-recurrent neural network (CNN-RNN), and a CNN-Transformer (CNN-Transformer), to incorporate both spatial and temporal information. Our ensemble model achieved F1 as 0.4133, accuracy as 0.6216 and final metric as 0.4821 on the validation set.

Motivation & Objective

• Address the challenge of facial expression recognition in unconstrained, real-world video environments with diverse backgrounds and lighting.
• Improve model generalization by leveraging pre-trained weights from large-scale face recognition datasets like VGGFace2.
• Effectively capture both spatial features from individual frames and temporal dynamics across sequential frames in video data.
• Develop a robust ensemble framework that combines multiple architectures to enhance performance on the ABAW 2021 facial expression recognition benchmark.

Proposed method

• Utilize a pre-trained ResNet50 model initialized with weights from the VGGFace2 dataset to improve feature extraction in unconstrained settings.
• Implement a CNN-RNN architecture where ResNet50 extracts spatial features from each frame, and a two-layer GRU processes the sequence to model temporal dependencies.
• Integrate a CNN-Transformer module that replaces RNNs with multi-head self-attention to capture long-range temporal relationships more efficiently.
• Apply sinusoidal positional encoding in the Transformer to preserve frame order information.
• Combine predictions from the three models (ResNet50, CNN-RNN, CNN-Transformer) via weighted averaging to form the final ensemble output.
• Train the models using SGD with a multi-step learning rate scheduler over 10 epochs on a 4-GPU setup, using 9-frame sequences for training in many-to-many mode and inference in many-to-one mode.

Experimental results

Research questions

• RQ1 How does pre-training on large-scale face recognition data improve performance in unconstrained facial expression recognition?
• RQ2 To what extent do RNN-based and Transformer-based temporal modeling approaches differ in capturing sequential dynamics in facial expression videos?
• RQ3 Can an ensemble of diverse architectures (CNN, CNN-RNN, CNN-Transformer) outperform individual models on the ABAW 2021 facial expression recognition benchmark?
• RQ4 How does the integration of spatial and temporal modeling components affect the final F1-score and combined evaluation metric?

Key findings

• The ensemble model achieved a final evaluation metric of 0.4821, representing a significant improvement over the baseline VGG-Face model (0.36).
• Pre-training ResNet50 on VGGFace2 improved the F1-score from 0.3763 to 0.4124 and accuracy from 0.5713 to 0.6216, demonstrating the benefit of large-scale pre-training.
• The CNN-RNN and CNN-Transformer components individually achieved F1-scores of 0.4126 and 0.4127, respectively, showing strong performance in temporal modeling.
• The full ensemble of all three models achieved the highest F1-score (0.4133), accuracy (0.6216), and final metric (0.4821), confirming the effectiveness of model ensembling.
• The pre-trained ResNet50 also showed strong performance on the AU detection track, achieving a final metric of 0.6734, indicating transferability of the learned features.