[Paper Review] Frame-level Prediction of Facial Expressions, Valence, Arousal and Action Units for Mobile Devices

Andrey V. Savchenko|arXiv (Cornell University)|Mar 25, 2022

Emotion and Mood RecognitionPsychology23 citations

TL;DR

Proposes a lightweight frame-level facial emotion analysis model using EfficientNet pretrained on AffectNet to predict expressions, valence/arousal, and action units, suitable for on-device mobile processing; demonstrates competitive results on Aff-Wild2 across ABAW3 challenges.

ABSTRACT

In this paper, we consider the problem of real-time video-based facial emotion analytics, namely, facial expression recognition, prediction of valence and arousal and detection of action unit points. We propose the novel frame-level emotion recognition algorithm by extracting facial features with the single EfficientNet model pre-trained on AffectNet. As a result, our approach may be implemented even for video analytics on mobile devices. Experimental results for the large scale Aff-Wild2 database from the third Affective Behavior Analysis in-the-wild (ABAW) Competition demonstrate that our simple model is significantly better when compared to the VggFace baseline. In particular, our method is characterized by 0.15-0.2 higher performance measures for validation sets in uni-task Expression Classification, Valence-Arousal Estimation and Expression Classification. Due to simplicity, our approach may be considered as a new baseline for all four sub-challenges.

Motivation & Objective

• Motivate real-time, on-device facial emotion analytics for mobile and embedded systems.
• Develop a single lightweight CNN-based pipeline that can perform multiple affective tasks without ensembles.
• leverage pre-trained facial representations to generalize across datasets and reduce computational burden.
• Show that a simple MLP-based head on top of EfficientNet features can achieve strong uni-task and multi-task performance.

Proposed method

• Pre-train a lightweight CNN on large face recognition data (VGGFace2) to learn generic facial features.
• Fine-tune the CNN on AffectNet for eight basic expressions to form an emotional feature extractor.
• Extract frame-level embeddings and expression scores from the fine-tuned network for each video frame.
• Train shallow MLP-based classifiers/regressors (one per task) using embeddings and/or scores as features.
• Optionally apply frame smoothing with mean or median filters across a sliding window to improve stability.

Experimental results

Research questions

• RQ1 Can a single lightweight model based on EfficientNet serve all four ABAW3 sub-challenges (FER, AU, Valence-Arousal) effectively in frame-level, on-device settings?
• RQ2 Do embeddings outperform emotion scores as features, and is concatenation of both beneficial for multi-task performance?
• RQ3 What is the impact of smoothing on frame-level predictions for each task?
• RQ4 How does a simple MLP head compare to more complex multi-task networks in multi-task learning for affective prediction?

Key findings

• A single EfficientNet-based feature extractor with a simple MLP head can outperform VGGFace-based baselines on ABAW3 tasks.
• Embeddings generally yield better expression/ AU predictions than emotion scores, and concatenating embeddings with scores provides strong performance.
• Frame-level smoothing with larger windows (e.g., k=15) yields notable improvements in Valence/Arousal and AU metrics.
• The proposed approach achieved competitive results, with mean improvements over baselines in several metrics across expression, AU, and VA tasks.
• In multi-task learning, EfficientNet-B0 with a simple logistic regression head performed best overall on the validation/test metrics, and the approach placed well among challenge entries.