[Paper Review] Multi-label Relation Modeling in Facial Action Units Detection

Xianpeng Ji, Yu Ding+3 more|arXiv (Cornell University)|Feb 4, 2020

Emotion and Mood Recognition7 references4 citations

TL;DR

This paper proposes a multi-label relation modeling approach for facial action unit (AU) detection using fused static texture and dynamic landmark features. By combining a GRU-based feature extraction network with a query-learned AU embedding table in a recurrent classification framework, the method achieves a validation F1-weighted accuracy of 0.56 on the Aff-Wild2 dataset, effectively modeling co-occurring AUs as sequential patterns.

ABSTRACT

This paper describes an approach to the facial action units detections. The involved action units (AU) include AU1 (Inner Brow Raiser), AU2 (Outer Brow Raiser), AU4 (Brow Lowerer), AU6 (Cheek Raise), AU12 (Lip Corner Puller), AU15 (Lip Corner Depressor), AU20 (Lip Stretcher), and AU25 (Lip Part). Our work relies on the dataset released by the FG-2020 Competition: Affective Behavior Analysis In-the-Wild (ABAW). The proposed method consists of the data preprocessing, the feature extraction and the AU classification. The data preprocessing includes the detection of face texture and landmarks. The texture static and landmark dynamic features are extracted through neural networks and then fused as the feature latent representation. Finally, the fused feature is taken as the initial hidden state of a recurrent neural network with a trainable lookup AU table. The output of the RNN is the results of AU classification. The detected accuracy is evaluated with 0.5$ imes$accuracy + 0.5$ imes$F1. Our method achieve 0.56 with the validation data that is specified by the organization committee.

Motivation & Objective

• To improve facial action unit detection by modeling co-occurring AUs as sequential patterns.
• To enhance feature representation by fusing static texture and dynamic landmark-based motion features.
• To develop a trainable AU embedding lookup mechanism that enables joint learning of AU presence across multiple labels.
• To address the challenge of multi-label, multi-class AU detection in real-world, unconstrained video data.
• To achieve robust performance on the AffectNet and Aff-Wild2 datasets using end-to-end learning with post-processing smoothing.

Proposed method

• A pre-trained face semantic segmentation model crops the face region and masks the background, followed by grayscale and edge image conversion for texture feature extraction.
• A dual-input CNN processes the grayscale and edge images to capture local AU-region correlations, with the final feature map passed to a GRU to model global texture context.
• Dynamic features are extracted from the difference between facial landmarks at adjacent time frames using a fully connected network with a Tanh activation to normalize scale.
• Static and dynamic features are fused via concatenation to form a unified latent representation for AU classification.
• A query-based GRU network uses learnable AU embeddings (8×64 matrix) as queries to iteratively classify each AU, with the GRU state updated per AU query.
• A final binary classifier (softmax) predicts AU activation based on the GRU output for each AU, trained end-to-end with joint optimization.

Experimental results

Research questions

• RQ1 How can static and dynamic facial features be effectively fused to improve multi-label AU detection?
• RQ2 Can modeling AU co-occurrence as a sequential process improve classification performance in facial expression analysis?
• RQ3 To what extent does a learnable AU embedding table enhance multi-label classification in weakly supervised settings?
• RQ4 How does the integration of landmark motion and texture features impact robustness in unconstrained video data?
• RQ5 What is the performance gain of using a recurrent query mechanism over standard multi-label classification heads?

Key findings

• The proposed method achieves a validation score of 0.56 on the Aff-Wild2 dataset using the combined metric of 0.5×accuracy + 0.5×F1.
• The fusion of static texture features (from grayscale and edge images) and dynamic motion features (from landmark differences) improves AU detection robustness.
• The use of a learnable AU embedding table with a query-based GRU enables effective modeling of multi-label AU co-occurrence patterns.
• The end-to-end training framework with hyper-parameter tuning and post-processing smoothing leads to consistent performance gains.
• The method demonstrates strong generalization on real-world, in-the-wild facial video data, particularly for co-occurring AUs like AU1, AU2, and AU4.
• The GRU-based sequential processing of AU embeddings allows for iterative refinement of predictions, improving detection of subtle and overlapping facial actions.