[Paper Review] Attribute-Guided Face Generation Using Conditional CycleGAN

Yongyi Lu, Yu‐Wing Tai+1 more|arXiv (Cornell University)|May 28, 2017

Generative Adversarial Networks and Image Synthesis20 references22 citations

TL;DR

This paper proposes conditional CycleGAN for attribute- and identity-guided face generation, enabling high-resolution face synthesis from low-resolution inputs with user-specified attributes (e.g., gender, hair color) or identity features. By integrating conditional vectors and a face verification network, the model achieves photorealistic, identity-preserving results in super-resolution, face swapping, and frontalization without paired training data.

ABSTRACT

We are interested in attribute-guided face generation: given a low-res face input image, an attribute vector that can be extracted from a high-res image (attribute image), our new method generates a high-res face image for the low-res input that satisfies the given attributes. To address this problem, we condition the CycleGAN and propose conditional CycleGAN, which is designed to 1) handle unpaired training data because the training low/high-res and high-res attribute images may not necessarily align with each other, and to 2) allow easy control of the appearance of the generated face via the input attributes. We demonstrate impressive results on the attribute-guided conditional CycleGAN, which can synthesize realistic face images with appearance easily controlled by user-supplied attributes (e.g., gender, makeup, hair color, eyeglasses). Using the attribute image as identity to produce the corresponding conditional vector and by incorporating a face verification network, the attribute-guided network becomes the identity-guided conditional CycleGAN which produces impressive and interesting results on identity transfer. We demonstrate three applications on identity-guided conditional CycleGAN: identity-preserving face superresolution, face swapping, and frontal face generation, which consistently show the advantage of our new method.

Motivation & Objective

• To enable high-quality, attribute-controlled face generation from low-resolution inputs using unpaired training data.
• To address identity preservation in face super-resolution and face swapping without relying on paired data.
• To develop a conditional CycleGAN framework that supports flexible control via user-supplied attributes or identity embeddings.
• To demonstrate robust performance under pose variations and partial occlusion in identity transfer tasks.
• To enable end-to-end, intervention-free frontal face generation from side-view inputs.

Proposed method

• Modifies the adversarial loss in CycleGAN to include a conditional feature vector as input to both generator and discriminator networks.
• Uses a pre-trained Light-CNN network to extract identity feature vectors for use as conditional inputs in identity-guided generation.
• Incorporates a face verification network as an auxiliary discriminator to enforce identity consistency during training.
• Leverages cycle consistency loss to learn bijective mappings between domains without requiring paired training samples.
• Applies the model to three applications: identity-preserving super-resolution, face swapping, and frontal face generation.
• Employs linear interpolation of conditional vectors to generate smooth transitions between facial attributes or identities.

Experimental results

Research questions

• RQ1 Can a conditional CycleGAN framework generate high-resolution face images with precise attribute control while preserving identity from low-resolution inputs?
• RQ2 How effectively can the model transfer facial identity across different poses and under partial occlusion?
• RQ3 Does incorporating a face verification loss improve the photorealism and identity fidelity in face-swapped results?
• RQ4 Can the model generate realistic frontal faces from non-frontal, low-resolution inputs in an end-to-end manner?
• RQ5 How well does the model generalize to unseen attribute or identity combinations through conditional vector interpolation?

Key findings

• The model achieves photorealistic face generation in identity-preserving super-resolution, with results that maintain the identity of the input even when the low-resolution image is from a different person.
• Face swapping results demonstrate high realism and accurate transfer of facial features like eyes, eyebrows, and hair, without requiring manual landmark detection or blending.
• The inclusion of face verification loss in the auxiliary discriminator leads to perceptually superior results, particularly in refining subtle features like eyebrows and eye shape.
• Linear interpolation of conditional vectors produces visually plausible, smooth transitions between facial attributes or identities, indicating effective generalization beyond memorization.
• Frontal face generation from side-view inputs is achieved end-to-end without human intervention, outperforming traditional methods that rely on warping and blending.
• The model performs robustly under pose variations and partial occlusion, demonstrating strong identity preservation in challenging conditions.