[Paper Review] Pipeline Generative Adversarial Networks for Facial Images Generation with Multiple Attributes

Ziqiang Zheng, Zhibin Yu+3 more|arXiv (Cornell University)|Nov 29, 2017

Generative Adversarial Networks and Image Synthesis30 references3 citations

TL;DR

This paper proposes Pipeline GAN (Pip-GAN), a two-stage generative adversarial network that generates high-fidelity facial images with multiple attributes—specifically head pose and facial expression—by sequentially applying a pose generator followed by an expression generator. The method improves image quality through a cascade loss, gradient penalty, and parallel classification, outperforming single-GAN and alternative pipeline structures on the KDEF dataset with clearer facial details and better fidelity.

ABSTRACT

Generative Adversarial Networks are proved to be efficient on various kinds of image generation tasks. However, it is still a challenge if we want to generate images precisely. Many researchers focus on how to generate images with one attribute. But image generation under multiple attributes is still a tough work. In this paper, we try to generate a variety of face images under multiple constraints using a pipeline process. The Pip-GAN (Pipeline Generative Adversarial Network) we present employs a pipeline network structure which can generate a complex facial image step by step using a neutral face image. We applied our method on two face image databases and demonstrate its ability to generate convincing novel images of unseen identities under multiple conditions previously.

Motivation & Objective

• To address the challenge of generating facial images with multiple attributes (e.g., pose and expression) simultaneously, which is difficult for standard GANs.
• To improve image generation quality by decomposing the complex task into two manageable stages: pose first, then expression.
• To enhance feature learning and generator control by introducing conditional vectors at the coded layer and a parallel classification discriminator.
• To evaluate the effectiveness of the pipeline structure (PE vs. EP) and ablation components (gradient penalty, cascade loss) on image fidelity and detail.
• To provide a robust solution for data augmentation in facial image analysis tasks requiring diverse attribute combinations.

Proposed method

• The model uses a two-stage pipeline: first, a pose generator (Gp) transforms a neutral face into different poses; second, an expression generator (Ge) modifies the pose image to apply a target facial expression.
• Each generator is paired with a dedicated discriminator (Dp for pose, De for expression), forming two separate GAN branches in sequence.
• Conditional vectors are injected at the latent code layer of each generator to control specific attributes (pose or expression) and guide generation precisely.
• A parallel classification discriminator is added at the bottleneck of the encoder-decoder network to enhance feature extraction and disentangle relevant attributes.
• The framework integrates cascade loss and gradient penalty to stabilize training and improve image quality, with the full loss function combining adversarial, reconstruction, and regularization terms.
• Two pipeline sequences are evaluated: PE (pose first) and EP (expression first), with ablation studies comparing performance across structures and components.

Experimental results

Research questions

• RQ1 Can a two-stage GAN pipeline effectively generate facial images with multiple attributes (e.g., pose and expression) better than a single GAN?
• RQ2 Does the order of attribute application (pose first vs. expression first) significantly affect image quality and detail fidelity?
• RQ3 How do additional components like gradient penalty and cascade loss improve the performance of the pipeline GAN?
• RQ4 To what extent can the proposed model generate realistic, high-quality images of unseen identities under diverse attribute combinations?
• RQ5 Can the model serve as an effective data augmentation tool for facial image datasets with limited attribute diversity?

Key findings

• The PE (pose-first) pipeline structure outperforms the EP (expression-first) structure, producing clearer images with more accurate facial details such as eyes, teeth, and mouth.
• The proposed Pip-GAN with full components (PE + gradient penalty + cascade loss + parallel classification) achieves the best performance across all metrics, surpassing both single-GAN and baseline pipeline models.
• While Pix2pix achieved competitive PSNR, it performed poorly on MSE and R-MSE, indicating blurry and distorted outputs, especially in facial expression generation.
• The pipeline approach significantly improves image quality over single-GAN models, with the PE structure generating plausible facial features even under extreme poses and expressions.
• The integration of gradient penalty and cascade loss led to measurable improvements in image fidelity and reduced mode collapse.
• The model successfully generated 24 distinct facial images per input (across 5 poses and 7 expressions), demonstrating effective data augmentation potential on the KDEF dataset.