[Paper Review] GANimation: Anatomically-aware Facial Animation from a Single Image

Albert Pumarola, Antonio Agudo+3 more|arXiv (Cornell University)|Jul 24, 2018

Generative Adversarial Networks and Image Synthesis34 references41 citations

TL;DR

GANimation trains an AU-conditioned GAN to animate a single face image along a continuous, anatomically feasible expression manifold, using unsupervised training and attention to focus on facial regions. It can interpolate expressions and handle wild images.

ABSTRACT

Recent advances in Generative Adversarial Networks (GANs) have shown impressive results for task of facial expression synthesis. The most successful architecture is StarGAN, that conditions GANs generation process with images of a specific domain, namely a set of images of persons sharing the same expression. While effective, this approach can only generate a discrete number of expressions, determined by the content of the dataset. To address this limitation, in this paper, we introduce a novel GAN conditioning scheme based on Action Units (AU) annotations, which describes in a continuous manifold the anatomical facial movements defining a human expression. Our approach allows controlling the magnitude of activation of each AU and combine several of them. Additionally, we propose a fully unsupervised strategy to train the model, that only requires images annotated with their activated AUs, and exploit attention mechanisms that make our network robust to changing backgrounds and lighting conditions. Extensive evaluation show that our approach goes beyond competing conditional generators both in the capability to synthesize a much wider range of expressions ruled by anatomically feasible muscle movements, as in the capacity of dealing with images in the wild.

Motivation & Objective

• Motivate continuous, anatomically feasible facial expression synthesis beyond discrete emotion categories.
• Enable expression control via Action Units (AUs) and interpolate between expressions.
• Develop an unsupervised training framework that does not require paired images of the same person under different expressions.
• Incorporate attention mechanisms to robustly handle background and lighting variations in wild images.

Proposed method

• Condition the GAN on a vector of action-unit activations (continuous magnitudes).
• Use a generator that outputs an attention mask A and a color transformation mask C, applying I_yf = (1−A)·C + A·I_yo.
• Train with a bidirectional rendering: map I_yo to I_yf and back to I_yo, enabling cycle-like consistency.
• Adopt a WGAN-GP critic with an auxiliary head to regress AU activations for conditioning fidelity.
• Include losses: image adversarial loss (WGAN-GP), attention smoothness via total variation, conditional AU regression loss, and identity (cycle) loss.
• Implement an attention-guided architecture to focus changes on facial regions while preserving non-face content.

Experimental results

Research questions

• RQ1 Can continuous AU conditioning enable a wider and more anatomically plausible range of expressions than discrete emotion categories?
• RQ2 Can an unsupervised AU-conditioned GAN render realistic expressions from a single input image without paired training data?
• RQ3 Does incorporating an attention mechanism improve realism and background/lighting robustness in the wild?
• RQ4 Can the model smoothly interpolate between expressions and apply edits to non-frontal or in-the-wild images?

Key findings

• The model can render anatomically coherent facial expressions by varying AU magnitudes, including smooth transitions and interpolation between expressions.
• Attention mechanism focuses changes on facial regions and preserves backgrounds, enabling high-quality results on wild images.
• Compared to DIAT, CycleGAN, IcGAN, and StarGAN on discrete emotion editing, GANimation achieves better visual quality and higher resolution with fewer artifacts.
• The approach handles background illumination changes and non-cropped faces by cropping to the face region and seamlessly merging with the original image via attention.
• Experiments show the ability to generate a wide range of expressions using as few as 14 AUs, demonstrating rich expressiveness and controllability.