[Paper Review] Facial Affective Behavior Analysis Method for 5th ABAW Competition

Shangfei Wang, Yanan Chang+6 more|arXiv (Cornell University)|Mar 16, 2023

Emotion and Mood Recognition8 citations

TL;DR

The paper builds three task-specific models for the 5th ABAW Competition on Aff-Wild2: VA estimation using fused backbones, expression classification with ensemble Sub- and Meta- Classifiers, and AU recognition combining CNN/Transformer with temporal processing and focal loss.

ABSTRACT

Facial affective behavior analysis is important for human-computer interaction. 5th ABAW competition includes three challenges from Aff-Wild2 database. Three common facial affective analysis tasks are involved, i.e. valence-arousal estimation, expression classification, action unit recognition. For the three challenges, we construct three different models to solve the corresponding problems to improve the results, such as data unbalance and data noise. For the experiments of three challenges, we train the models on the provided training data and validate the models on the validation data.

Motivation & Objective

• Motivate automatic analysis of facial affective behavior for human-computer interaction in-the-wild.
• Address three ABAW challenges (valence-arousal estimation, expression classification, action unit recognition).
• Develop task-specific models to handle data imbalance and noise in Aff-Wild2.
• Evaluate models on provided training/validation sets and report performance gains over baselines.

Proposed method

• For VA estimation: fuse features from ResNet50, RegNet, and EfficientNet backbones; use polarity classifiers before valence/arousal classifiers; input valence features to arousal classifier; train with cross-entropy on discretized labels and CCC loss for regression.
• For expression classification: apply bagging ensemble to train multiple sub-classifiers on augmented data; a Meta-classifier performs final voting with a predefined rule and threshold; use L_C = L_CE + lambda * L_Dice with focal data augmentation strategies; data deletion/handling for class 7.
• For AU recognition: preprocess by face Cropping/Alignment; extract features with ResNet-50 and RegNet backbones; employ temporal modeling via Transformer blocks and up/down-sampling pipelines; fuse three pipeline outputs; train with focal loss to handle imbalance.

Experimental results

Research questions

• RQ1 How to improve valence-arousal estimation in-the-wild via backbone fusion and cross-task correlations?
• RQ2 Can ensemble and meta-classification strategies improve facial expression recognition under data imbalance?
• RQ3 How to effectively model temporal and spatial cues for AU detection using CNN backbones and Transformer-based temporal modules?

Key findings

• Valence and arousal CCC on validation: baseline 22.0 (valence) and 24.0 (arousal); proposed method 25.7 (valence) and 38.3 (arousal).
• Expression classification: Meta-Classifier achieves best validation metrics (Acc 0.4618, F1 0.3024) among evaluated components.
• AU detection: RegNet-800MF backbone yields the highest validation F1 score of 0.6983, outperforming RegNet-400MF (0.6813) and ResNet-50 (0.5591).