[Paper Review] StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks

Han Zhang, Tao Xu+5 more|arXiv (Cornell University)|Dec 9, 2016

Generative Adversarial Networks and Image SynthesisComputer Science227 citations

TL;DR

StackGAN generates 256x256 photo-realistic images from text by decomposing the task into Stage-I (sketch) and Stage-II (refinement) GANs, with Conditioning Augmentation to improve diversity and stability.

ABSTRACT

Synthesizing high-quality images from text descriptions is a challenging problem in computer vision and has many practical applications. Samples generated by existing text-to-image approaches can roughly reflect the meaning of the given descriptions, but they fail to contain necessary details and vivid object parts. In this paper, we propose Stacked Generative Adversarial Networks (StackGAN) to generate 256x256 photo-realistic images conditioned on text descriptions. We decompose the hard problem into more manageable sub-problems through a sketch-refinement process. The Stage-I GAN sketches the primitive shape and colors of the object based on the given text description, yielding Stage-I low-resolution images. The Stage-II GAN takes Stage-I results and text descriptions as inputs, and generates high-resolution images with photo-realistic details. It is able to rectify defects in Stage-I results and add compelling details with the refinement process. To improve the diversity of the synthesized images and stabilize the training of the conditional-GAN, we introduce a novel Conditioning Augmentation technique that encourages smoothness in the latent conditioning manifold. Extensive experiments and comparisons with state-of-the-arts on benchmark datasets demonstrate that the proposed method achieves significant improvements on generating photo-realistic images conditioned on text descriptions.

Motivation & Objective

• Motivate and address the difficulty of generating high-resolution, photo-realistic images from text descriptions.
• Decompose text-to-image synthesis into two manageable stages to improve detail and fidelity.
• Stabilize training and increase diversity through Conditioning Augmentation (CA).
• Demonstrate quantitative and qualitative improvements over prior text-to-image methods on standard datasets.

Proposed method

• Propose StackGAN architecture with Stage-I generating a low-resolution sketch conditioned on text.
• Stage-II refines Stage-I outputs and adds details to produce high-resolution images, conditioned on both Stage-I results and text.
• Introduce Conditioning Augmentation to sample stochastic conditioning variables from a Gaussian parameterized by text embeddings, plus a KL-divergence regularization term.
• Use a matching-aware discriminator to better align images with text descriptions across both stages.
• Train via staged adversarial losses for Stage-I and Stage-II, using ADAM and standard GAN training procedures.

Experimental results

Research questions

• RQ1 Can a two-stage stacked GAN framework produce higher-resolution, more realistic images conditioned on text than single-stage approaches?
• RQ2 Does Conditioning Augmentation improve diversity and stabilize training for conditional GANs in text-to-image synthesis?
• RQ3 How does Stage-II refinement correct Stage-I defects and add details to achieve 256x256 realism?

Key findings

• StackGAN achieves 256x256 photo-realistic images conditioned on text descriptions, outperforming state-of-the-art methods on multiple datasets.
• conditioning augmentation improves training stability and sample diversity, evidenced by higher inception scores and diverse outputs.
• Stage-II refinement consistently improves image quality by correcting Stage-I defects and adding text-consistent details.
• Inception scores and human evaluation favor StackGAN over GAN-INT-CLS and GAWWN across CUB, Oxford-102, and COCO.
• Stage-I alone struggles to produce plausible high-resolution images, while StackGAN's two-stage design yields superior results.