[Paper Review] Geometry-Contrastive GAN for Facial Expression Transfer

Fengchun Qiao, Naiming Yao+4 more|arXiv (Cornell University)|Feb 6, 2018

Face recognition and analysis24 references51 citations

TL;DR

GC-GAN learns a continuous semantic latent space for facial expressions by embedding geometry with contrastive learning, enabling identity-preserving expression transfer across different subjects.

ABSTRACT

In this paper, we propose a Geometry-Contrastive Generative Adversarial Network (GC-GAN) for transferring continuous emotions across different subjects. Given an input face with certain emotion and a target facial expression from another subject, GC-GAN can generate an identity-preserving face with the target expression. Geometry information is introduced into cGANs as continuous conditions to guide the generation of facial expressions. In order to handle the misalignment across different subjects or emotions, contrastive learning is used to transform geometry manifold into an embedded semantic manifold of facial expressions. Therefore, the embedded geometry is injected into the latent space of GANs and control the emotion generation effectively. Experimental results demonstrate that our proposed method can be applied in facial expression transfer even there exist big differences in facial shapes and expressions between different subjects.

Motivation & Objective

• Motivate facial expression transfer across subjects with continuous emotion representations.
• Introduce geometry as a continuous condition to guide expression synthesis in GANs.
• Embed facial geometry into a semantic latent space using contrastive learning to reduce misalignment across subjects.
• Ensure Lipschitz continuity of the generator with respect to expression-conditioning latent variables.
• Demonstrate effectiveness on multiple datasets and through cross-dataset transfer scenarios.

Proposed method

• Introduce a three-component GC-GAN: a geometry embedding network E, an image generator G, and an image discriminator D.
• Embed geometry using contrastive learning to obtain z_g = E_enc(g_k^v), forming a semantic expression manifold.
• Concatenate identity latent z_i (from I_j^u) with geometry latent z_g to form the generator input, enforcing Lipschitz continuity.
• Train E with a contrastive loss L_contr to pull same-expression landmarks together and push different expressions apart.
• Use adversarial losses (WGAN-GP) plus reconstruction losses (L_ir for images, L_gr for landmarks) to train G and E.
• Proceed in two stages: pretrain E with L_contr and L_gr, then train G and D with fixed E, optimizing L_ir and L_adv.

Experimental results

Research questions

• RQ1 Can continuous geometry embeddings guide expression transfer across subjects while preserving identity?
• RQ2 Does contrastive embedding of facial geometry yield a semantic, Lipschitz-continuous expression latent space that improves transfer quality?
• RQ3 How does GC-GAN perform across diverse datasets and in cross-dataset expression transfer scenarios?
• RQ4 What is the impact of the reconstruction and adversarial losses on synthesis quality and expression fidelity?

Key findings

• GC-GAN enables identity-preserving facial expression transfer guided by an embedded geometry latent space.
• Incorporating L_ir, L_gr, and L_contr improves SSIM and PSNR over ablations and a baselines like CDAAE across Multi-PIE, CK+, and BU-4DFE.
• The geometry embedding yields a Lipschitz-continuous mapping with respect to expression conditions, facilitating smooth expression transitions.
• Cross-dataset experiments (CelebA guiding CelebA-derived landmarks) show GC-GAN can transfer expressions while preserving identity.
• Ablations show removing any of the three losses degrades performance, highlighting the contribution of reconstruction and contrastive guidance.