[Paper Review] Pose-aware Adversarial Domain Adaptation for Personalized Facial Expression Recognition

Guang Liang, Shangfei Wang+1 more|arXiv (Cornell University)|Jul 12, 2020

Domain Adaptation and Few-Shot Learning25 references4 citations

TL;DR

This paper proposes a novel unsupervised adversarial domain adaptation method for personalized facial expression recognition that jointly addresses pose and subject variations. By integrating adversarial domain adaptation, cross adversarial feature disentanglement, and reconstruction learning, the model learns pose- and identity-robust expression features without relying on synthetic images, achieving state-of-the-art performance on multiple benchmarks including SFEW and AffectNet.

ABSTRACT

Current facial expression recognition methods fail to simultaneously cope with pose and subject variations. In this paper, we propose a novel unsupervised adversarial domain adaptation method which can alleviate both variations at the same time. Specially, our method consists of three learning strategies: adversarial domain adaptation learning, cross adversarial feature learning, and reconstruction learning. The first aims to learn pose- and expression-related feature representations in the source domain and adapt both feature distributions to that of the target domain by imposing adversarial learning. By using personalized adversarial domain adaptation, this learning strategy can alleviate subject variations and exploit information from the source domain to help learning in the target domain. The second serves to perform feature disentanglement between pose- and expression-related feature representations by impulsing pose-related feature representations expression-undistinguished and the expression-related feature representations pose-undistinguished. The last can further boost feature learning by applying face image reconstructions so that the learned expression-related feature representations are more pose- and identity-robust. Experimental results on four benchmark datasets demonstrate the effectiveness of the proposed method.

Motivation & Objective

• To address the dual challenge of pose variation and subject (identity) variation in facial expression recognition.
• To develop an unsupervised domain adaptation framework that transfers knowledge from a source domain (labeled data) to a target domain (unlabeled, novel subjects) without requiring labeled target samples.
• To disentangle pose-related and expression-related features to improve robustness to pose and identity changes.
• To enhance feature representation quality through reconstruction learning, ensuring expression features are invariant to pose and identity.
• To outperform existing methods in multi-pose, in-the-wild facial expression recognition settings.

Proposed method

• Employs adversarial domain adaptation with separate domain discriminators for pose and expression features to align source and target domain distributions.
• Introduces a cross adversarial learning strategy where a pose discriminator is trained to be unable to distinguish expression features, and an expression classifier is unable to distinguish pose features, enabling feature disentanglement.
• Uses reconstruction learning by combining cross-domain pose and expression features to reconstruct original face images, enforcing robustness in learned representations.
• Applies joint training of three learning strategies: adversarial domain adaptation, cross adversarial feature learning, and reconstruction learning.
• Utilizes separate encoders for source and target domains, with the source encoder supervised on labeled data and the target encoder trained via adversarial objectives.
• Employs a generator network to reconstruct face images from disentangled features, reinforcing the quality and disentanglement of the learned representations.

Experimental results

Research questions

• RQ1 Can an unsupervised domain adaptation framework effectively reduce the domain shift caused by both pose and subject variations in facial expression recognition?
• RQ2 How can pose- and expression-related features be disentangled in a way that enhances robustness to pose and identity changes?
• RQ3 Can reconstruction-based learning improve the robustness of expression features to pose and identity variations without relying on image generation?
• RQ4 Does the proposed method outperform existing state-of-the-art approaches in multi-pose, in-the-wild facial expression recognition benchmarks?
• RQ5 How does the method perform under limited labeled data in the target domain, particularly when compared to methods relying on synthetic data or data augmentation?

Key findings

• On the SFEW dataset, the proposed method achieves an average accuracy of 31.3%, outperforming the next best method (UPADA R+adv+cross) at 30.9%.
• When trained on AffectNet and tested on SFEW, the method achieves 57.6% average accuracy, surpassing the previous state-of-the-art (UPADA R+adv+cross) at 56.3%.
• On the AffectNet test set, the method achieves 59.0% average accuracy, exceeding the next best method (UPADA R+adv+cross) at 58.3%.
• The method shows superior performance on rare or imbalanced expression categories, such as surprise (SU), where it achieves 29.8% on SFEW (AffectNet training), compared to IPFR’s 31.4% but with better class-wise balance.
• The ablation study confirms that all three components—adversarial adaptation, cross adversarial learning, and reconstruction—contribute significantly to performance, with reconstruction providing the largest gain.
• The method outperforms GAN-based methods like GARN and IA-gen, which suffer from noise introduced by unrealistic generated images, by learning robust features directly from real data.