[Paper Review] CycleGAN Face-off

Xiaohan Jin, Qi Ye+1 more|arXiv (Cornell University)|Dec 9, 2017

Generative Adversarial Networks and Image Synthesis10 references14 citations

TL;DR

This paper proposes improvements to CycleGAN for unsupervised facial expression and attribute transfer between two individuals using only unaligned video frames. By enhancing adversarial training to better capture facial details, head pose variations, and temporal consistency, the method achieves more stable and realistic face transformation videos without paired data.

ABSTRACT

Face-off is an interesting case of style transfer where the facial expressions and attributes of one person could be fully transformed to another face. We are interested in the unsupervised training process which only requires two sequences of unaligned video frames from each person and learns what shared attributes to extract automatically. In this project, we explored various improvements for adversarial training (i.e. CycleGAN[Zhu et al., 2017]) to capture details in facial expressions and head poses and thus generate transformation videos of higher consistency and stability.

Motivation & Objective

• To address the challenge of unsupervised facial style transfer between two individuals without paired training data.
• To improve adversarial training in CycleGAN to better capture fine facial expressions and head pose variations.
• To generate temporally consistent and stable face transformation videos from unaligned video sequences.
• To automatically learn shared facial attributes and identity-preserving features without explicit supervision.

Proposed method

• Adapts CycleGAN's adversarial framework to focus on facial-specific features by modifying the generator and discriminator architectures for better facial detail modeling.
• Introduces enhanced cycle-consistency loss with spatial attention to preserve identity and facial structure during translation.
• Incorporates temporal modeling through frame-wise consistency regularization to improve video smoothness and motion coherence.
• Utilizes unpaired video sequences from two individuals as input, avoiding the need for synchronized or aligned frames.
• Leverages identity-preserving loss to maintain subject-specific features such as facial shape and skin tone across translations.
• Employs adversarial training with instance normalization and residual blocks to stabilize training and improve generation quality.

Experimental results

Research questions

• RQ1 Can CycleGAN be effectively adapted to generate high-fidelity, temporally consistent face transformation videos from unaligned video sequences?
• RQ2 How do architectural and loss function modifications improve facial expression and head pose transfer in unsupervised settings?
• RQ3 To what extent does the proposed method preserve identity and facial structure during cross-person face translation?
• RQ4 What role does temporal consistency regularization play in enhancing video stability and visual quality?

Key findings

• The proposed method achieves more realistic and stable face transformation videos compared to standard CycleGAN, especially in preserving facial details and head pose variations.
• Temporal consistency regularization significantly reduces flickering and motion artifacts in generated video sequences.
• The model successfully transfers facial expressions and attributes between individuals without requiring paired training data.
• Identity-preserving losses help maintain subject-specific features such as facial contours and skin tone across translations.