[Paper Review] AU-Guided Unsupervised Domain Adaptive Facial Expression Recognition

Kai Wang, Yuxin Gu+4 more|arXiv (Cornell University)|Dec 18, 2020

Emotion and Mood Recognition52 references4 citations

TL;DR

This paper proposes AdaFER, an AU-guided unsupervised domain adaptation framework for facial expression recognition that leverages facial action units (AUs) to reduce annotation bias between datasets. By using AU-based soft label assignment and AU-guided triplet loss, AdaFER achieves state-of-the-art performance on multiple benchmarks, including 81.40% accuracy on CK+ and 70.86% on ExpW.

ABSTRACT

The domain diversities including inconsistent annotation and varied image collection conditions inevitably exist among different facial expression recognition (FER) datasets, which pose an evident challenge for adapting the FER model trained on one dataset to another one. Recent works mainly focus on domain-invariant deep feature learning with adversarial learning mechanism, ignoring the sibling facial action unit (AU) detection task which has obtained great progress. Considering AUs objectively determine facial expressions, this paper proposes an AU-guided unsupervised Domain Adaptive FER (AdaFER) framework to relieve the annotation bias between different FER datasets. In AdaFER, we first leverage an advanced model for AU detection on both source and target domain. Then, we compare the AU results to perform AU-guided annotating, i.e., target faces that own the same AUs with source faces would inherit the labels from source domain. Meanwhile, to achieve domain-invariant compact features, we utilize an AU-guided triplet training which randomly collects anchor-positive-negative triplets on both domains with AUs. We conduct extensive experiments on several popular benchmarks and show that AdaFER achieves state-of-the-art results on all these benchmarks.

Motivation & Objective

• Address the domain shift problem in facial expression recognition caused by inconsistent annotations and varying image collection conditions across datasets.
• Reduce annotation bias between source and target FER datasets by leveraging the objective, low-bias nature of facial action units (AUs).
• Improve feature compactness and domain invariance in unsupervised domain adaptation by integrating AU-based supervision into the training process.
• Achieve state-of-the-art performance on multiple cross-dataset FER benchmarks without increasing inference cost.

Proposed method

• Apply a pretrained AU detection model to extract action unit codes from both source and target domain images.
• Perform AU-guided annotating by retrieving source-domain images with matching AU codes to assign soft labels to target-domain images.
• Construct AU-guided triplet samples by pairing source images (anchors) with target images that share or differ in AU codes (positive/negative samples).
• Train the model using a triplet loss that encourages compact, domain-invariant features by minimizing intra-class and maximizing inter-class distances based on AU similarity.
• Fine-tune a source-pretrained FER model on the target domain using both pseudo soft labels and AU-guided triplet loss.
• Optimize the triplet sampling threshold to balance difficulty and training stability, with ablation showing optimal performance at 0.5.

Experimental results

Research questions

• RQ1 Can facial action units (AUs) serve as a reliable auxiliary signal to mitigate annotation bias in cross-domain facial expression recognition?
• RQ2 How effective is AU-guided soft labeling in improving generalization when no ground-truth labels are available on the target domain?
• RQ3 To what extent does AU-guided triplet training enhance feature compactness and domain invariance in unsupervised domain adaptation?
• RQ4 Does integrating AU-based supervision lead to state-of-the-art performance across diverse FER benchmarks?

Key findings

• AdaFER achieves a new state-of-the-art accuracy of 81.40% on the CK+ dataset in cross-dataset facial expression recognition.
• On the ExpW benchmark, AdaFER achieves 70.86% accuracy, significantly outperforming prior methods and setting a new SOTA.
• The method improves baseline performance by 8.94% in mean accuracy across all target datasets when using RAF-DB as the source.
• Feature visualization via T-SNE confirms that AdaFER learns more compact and well-separated features than the baseline, especially for neutral expressions.
• The ablation study shows that setting the triplet mining threshold to 0.5 yields optimal performance, balancing sample difficulty and training stability.
• AdaFER achieves these gains without increasing inference cost, unlike methods requiring complex inference-time feature fusion or node initialization.