[Paper Review] Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Network

Zizhao Zhang, Yuanpu Xie+1 more|arXiv (Cornell University)|Feb 26, 2018

Generative Adversarial Networks and Image SynthesisComputer Science44 references52 citations

TL;DR

HDGAN introduces hierarchically-nested discriminators at multiple generator scales to produce high-resolution, semantically consistent photographic images conditioned on text. It improves state-of-the-art across datasets and metrics, with a new visual-semantic similarity evaluation.

ABSTRACT

This paper presents a novel method to deal with the challenging task of generating photographic images conditioned on semantic image descriptions. Our method introduces accompanying hierarchical-nested adversarial objectives inside the network hierarchies, which regularize mid-level representations and assist generator training to capture the complex image statistics. We present an extensile single-stream generator architecture to better adapt the jointed discriminators and push generated images up to high resolutions. We adopt a multi-purpose adversarial loss to encourage more effective image and text information usage in order to improve the semantic consistency and image fidelity simultaneously. Furthermore, we introduce a new visual-semantic similarity measure to evaluate the semantic consistency of generated images. With extensive experimental validation on three public datasets, our method significantly improves previous state of the arts on all datasets over different evaluation metrics.

Motivation & Objective

• Address the challenge of generating high-resolution, photorealistic images conditioned on descriptive text.
• Regularize mid-level generator representations using hierarchical adversarial constraints.
• Enable end-to-end training of a single-stream generator with multi-scale discriminators.
• Improve semantic consistency and image fidelity through multi-purpose adversarial losses.
• Introduce a visual-semantic similarity metric to evaluate alignment between text and generated images.

Proposed method

• Use a single-stream generator that outputs an image pyramid with side outputs at multiple resolutions.
• Attach a distinct discriminator to each side output to enforce real/fake and image-text pair discrimination at corresponding scales.
• Employ hierarchical-nested adversarial objectives to regularize intermediate representations and stabilize training.
• Adopt matching-aware pair losses for global semantic consistency and local adversarial losses for local fidelity at each scale.
• Incorporate conditioning augmentation with a stochastic text embedding and KL regularization to diversify conditioning.
• Train with a multi-term loss that includes both image-text pair losses and local image losses across scales.

Experimental results

Research questions

• RQ1 How can hierarchical, multi-scale adversarial supervision improve high-resolution text-to-image synthesis?
• RQ2 Does injecting local as well as global adversarial cues at multiple generator levels yield better semantic fidelity and visual quality?
• RQ3 Can a single-stream generator achieve high-resolution, semantically consistent results without multi-stage cascades or extra supervision?
• RQ4 What is the impact of a visual-semantic similarity measure for evaluating text-image alignment beyond traditional GAN metrics?

Key findings

• HDGAN achieves higher Inception scores than several baselines on CUB, Oxford-102, and COCO (e.g., 4.15 ± 0.05 on CUB, 3.45 ± 0.07 on Oxford-102, and 11.86 ± 0.18 on COCO).
• Hierarchically-nested discriminators improve performance; using discriminators at 64×64, 128×128, and 256×256 scales yields stronger results than fewer scales.
• Local image losses and a multi-scale, multi-loss framework improve visual-semantic consistency and generate more detailed structures (e.g., petals and bird poses).
• HDGAN outperforms StackGAN and TAC-GAN across several metrics, including visual-semantic similarity (VS) and MS-SSIM, and shows better sample diversity.
• A new visual-semantic similarity metric correlates with semantic alignment between generated images and conditioned text, outperforming some baselines.
• HDGAN demonstrates effective style transfer via sentence interpolation, indicating smooth latent space semantics.