[Paper Review] EmotiEffNet Facial Features in Uni-task Emotion Recognition in Video at ABAW-5 competition

Andrey V. Savchenko|arXiv (Cornell University)|Mar 16, 2023

Emotion and Mood RecognitionPsychology15 citations

TL;DR

The paper proposes a video emotion-recognition pipeline using EmotiEffNet embeddings with MLP/LightAutoML classifiers and temporal smoothing, achieving improved VA, FER, and AU metrics on Aff-Wild2 at ABAW-5.

ABSTRACT

In this article, the results of our team for the fifth Affective Behavior Analysis in-the-wild (ABAW) competition are presented. The usage of the pre-trained convolutional networks from the EmotiEffNet family for frame-level feature extraction is studied. In particular, we propose an ensemble of a multi-layered perceptron and the LightAutoML-based classifier. The post-processing by smoothing the results for sequential frames is implemented. Experimental results for the large-scale Aff-Wild2 database demonstrate that our model achieves a much greater macro-averaged F1-score for facial expression recognition and action unit detection and concordance correlation coefficients for valence/arousal estimation when compared to baseline.

Motivation & Objective

• Motivate improved frame-level emotion prediction in-the-wild for VA, FER, and AU tasks under ABAW-5 constraints.
• Leverage EmotiEffNet facial embeddings pretrained on AffectNet to produce frame-level representations independent of Aff-Wild2-specific biases.
• Evaluate simple yet effective classifiers (MLP and LightAutoML ensembles) with temporal smoothing on downstream tasks.
• Compare against baseline CNNs and prior EfficientNet-based approaches to quantify gains.
• Provide a reproducible workflow and discuss potential future enhancements like audio integration and sequential inference.

Proposed method

• Extract per-frame embeddings x(t) and logits l(t) from pre-trained EmotiEffNet models (EmotiEffNet-B0 or MT-EmotiEffNet-B0) finetuned for FER and VA tasks.
• Concatenate logits l(t), valence V(t), and arousal A(t) as inputs to classifiers (MLP and LightAutoML) to predict VA, FER class, or AU labels.
• For VA, train an MLP (no hidden layer) to maximize Concordance Correlation Coefficient for valence and arousal using tanh activations.
• For FER and AU, train MLPs with one hidden layer; use softmax for FER and sigmoid with thresholds for AU detection.
• Optionally finetune EmotiEffNet on the ABAW-5 data and apply frame-level smoothing via a box/median filter with kernel size k to improve stability.
• Experiment with blending/ensembles (LightAutoML, MLP, finetuned model) and pre-trained VA-only or pre-trained logits presets to boost performance.
• Smoothing kernel size k is a key hyperparameter; larger k (e.g., 25–50 for VA/AR) often yields better CCC, while smaller k is better for AU detection (3–5).
• Provide a reproducible workflow and publicly available training code (GitHub) for replication.

Experimental results

Research questions

• RQ1 Can EmotiEffNet-based facial features improve frame-level VA, FER, and AU performance on Aff-Wild2 in ABAW-5 compared to baselines?
• RQ2 What is the impact of classifier choice (MLP vs LightAutoML) and feature inputs (embeddings vs logits) on downstream task performance?
• RQ3 How does temporal smoothing affect prediction stability and accuracy across VA, FER, and AU tasks?
• RQ4 Does fine-tuning EmotiEffNet on ABAW-5 data provide measurable gains over using frozen pre-trained embeddings?
• RQ5 What is the benefit of blending multiple classifiers and input representations for uni-task ABAW-5 challenges?

Key findings

• The pipeline with EmotiEffNet embeddings and MLP/LightAutoML ensembles significantly improves VA CCC scores over baseline ResNet-50 and prior EfficientNet usage.
• FER macro F1-score and accuracy improve notably with smoothing and ensemble strategies, achieving substantial gains over the VGGFACE baseline.
• AU detection macro F1-score improves with MLP-based approaches and smoothing, outperforming several baselines and indicating benefits from temporal blending.
• Best VA results reached CCC_V and CCC_A improvements up to notable margins over baselines and previous EmotiEffNet configurations, with smoothing kernel sizes in the 25–50 range yielding strong CCC gains.
• AU results show that smaller smoothing (k around 3–5) is preferable due to rapid frame changes, and ensemble/threshold optimization further boosts F1 scores.
• Across tasks, the proposed EmotiEffNet-based workflow outperforms the official baselines and prior EfficientNet applications on validation data.