[Paper Review] From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos

Yin Chen, Jia Li+3 more|arXiv (Cornell University)|Dec 9, 2023

Emotion and Mood Recognition4 citations

TL;DR

This paper proposes S2D, a parameter-efficient framework that adapts pre-trained landmark-aware image models for dynamic facial expression recognition (DFER) in videos by integrating Temporal-Modeling Adapters (TMAs) and a novel Emotion-Anchors based Self-Distillation Loss (SDL). The method achieves state-of-the-art performance on DFEW with under 10% additional trainable parameters, significantly improving robustness to ambiguous labels and modeling temporal dynamics through landmark-aware features.

ABSTRACT

Dynamic facial expression recognition (DFER) in the wild is still hindered by data limitations, e.g., insufficient quantity and diversity of pose, occlusion and illumination, as well as the inherent ambiguity of facial expressions. In contrast, static facial expression recognition (SFER) currently shows much higher performance and can benefit from more abundant high-quality training data. Moreover, the appearance features and dynamic dependencies of DFER remain largely unexplored. To tackle these challenges, we introduce a novel Static-to-Dynamic model (S2D) that leverages existing SFER knowledge and dynamic information implicitly encoded in extracted facial landmark-aware features, thereby significantly improving DFER performance. Firstly, we build and train an image model for SFER, which incorporates a standard Vision Transformer (ViT) and Multi-View Complementary Prompters (MCPs) only. Then, we obtain our video model (i.e., S2D), for DFER, by inserting Temporal-Modeling Adapters (TMAs) into the image model. MCPs enhance facial expression features with landmark-aware features inferred by an off-the-shelf facial landmark detector. And the TMAs capture and model the relationships of dynamic changes in facial expressions, effectively extending the pre-trained image model for videos. Notably, MCPs and TMAs only increase a fraction of trainable parameters (less than +10\%) to the original image model. Moreover, we present a novel Emotion-Anchors (i.e., reference samples for each emotion category) based Self-Distillation Loss to reduce the detrimental influence of ambiguous emotion labels, further enhancing our S2D. Experiments conducted on popular SFER and DFER datasets show that we achieve the state of the art.

Motivation & Objective

• To address the performance gap between static (SFER) and dynamic (DFER) facial expression recognition due to limited, ambiguous, and less diverse video data.
• To leverage abundant, high-quality SFER datasets and pre-trained image models as prior knowledge for DFER.
• To effectively model temporal dynamics in facial expressions using lightweight adapters without retraining from scratch.
• To reduce uncertainty and ambiguity in emotion predictions through a novel self-distillation loss using emotion anchors.
• To develop a parameter-efficient, practical framework for DFER that outperforms existing methods with minimal computational overhead.

Proposed method

• A pre-trained Vision Transformer (ViT) with Multi-View Complementary Prompters (MCPs) is fine-tuned on SFER data to enhance landmark-aware appearance features.
• Temporal-Modeling Adapters (TMAs) are inserted into the image model to capture sequential dynamics in video clips, enabling end-to-end video-level learning.
• MCPs integrate facial landmark features from a pre-trained detector (MobileFaceNet) to enrich appearance representations with geometric facial structure.
• The Emotion-Anchors based Self-Distillation Loss (SDL) uses reference samples per emotion category to generate soft labels, reducing model uncertainty on ambiguous expressions.
• The S2D framework is trained end-to-end with only ~10% additional trainable parameters compared to the base image model.
• t-SNE and attention visualization are used to analyze feature discriminability and temporal attention patterns, validating the model's ability to capture dynamic changes.

Experimental results

Research questions

• RQ1 Can pre-trained landmark-aware image models be effectively adapted for video-based facial expression recognition with minimal parameter updates?
• RQ2 How can temporal dynamics in facial expressions be effectively modeled in a lightweight and efficient manner?
• RQ3 To what extent can self-distillation with emotion anchors reduce the impact of ambiguous emotion annotations in DFER datasets?
• RQ4 Can the integration of static SFER knowledge and dynamic video modeling lead to state-of-the-art performance on in-the-wild DFER benchmarks?
• RQ5 How does the proposed method compare in parameter efficiency and performance to existing SOTA methods in DFER?

Key findings

• S2D achieves a new state-of-the-art weighted average recall (WAR) of 78.9% on the DFEW test set, outperforming prior methods including MAE-DFER and DFER-CLIP.
• The model achieves this performance with only 9 million tunable parameters, representing less than 10% increase over the base image model.
• The inclusion of TMAs significantly improves feature discriminability, as shown by t-SNE visualization, with more compact and separable feature clusters.
• The SDL module reduces ambiguity in predictions: for example, it increases the confidence difference between 'happy' and 'neutral' from 0.02 to 0.23 in ambiguous cases.
• Attention visualization confirms that the model dynamically shifts focus across facial regions (eyes, mouth) over time, capturing muscle movement evolution.
• Despite data scarcity, the model shows improved performance on rare classes like 'disgust' and 'fear', though these remain challenging due to limited samples.