[Paper Review] Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis

Seunghoon Hong, Dingdong Yang+2 more|arXiv (Cornell University)|Jan 16, 2018

Generative Adversarial Networks and Image Synthesis30 references45 citations

TL;DR

The paper proposes a hierarchical text-to-image synthesis framework that first infers a semantic layout (bounding boxes and shapes) from text and then generates images conditioned on that layout, improving semantic alignment and interpretability on MS-COCO.

ABSTRACT

We propose a novel hierarchical approach for text-to-image synthesis by inferring semantic layout. Instead of learning a direct mapping from text to image, our algorithm decomposes the generation process into multiple steps, in which it first constructs a semantic layout from the text by the layout generator and converts the layout to an image by the image generator. The proposed layout generator progressively constructs a semantic layout in a coarse-to-fine manner by generating object bounding boxes and refining each box by estimating object shapes inside the box. The image generator synthesizes an image conditioned on the inferred semantic layout, which provides a useful semantic structure of an image matching with the text description. Our model not only generates semantically more meaningful images, but also allows automatic annotation of generated images and user-controlled generation process by modifying the generated scene layout. We demonstrate the capability of the proposed model on challenging MS-COCO dataset and show that the model can substantially improve the image quality, interpretability of output and semantic alignment to input text over existing approaches.

Motivation & Objective

• Motivate and address the challenge of generating complex, real-world images from general text descriptions.
• Introduce a semantic-layout-based generation pipeline to decouple text-to-layout and layout-to-image synthesis.
• Enable fine-grained control and automatic annotation by predicting object-level layouts.
• Demonstrate improved semantic fidelity and image quality over prior text-to-image methods on MS-COCO.

Proposed method

• Use a two-stage layout generator: a box generator that auto-regressively predicts object bounding boxes and labels from text, followed by a shape generator that refines each object’s mask inside its box.
• Construct a semantic label map by aggregating instance masks, which guides the image generator.
• Train box and shape generators with supervised losses (NLL for boxes and labels; adversarial and perceptual losses for masks) to enforce instance-wise and global layout consistency.
• Generate images conditioned on both the inferred semantic layout and the input text using a cascaded encoder-decoder with attention over text, plus a discriminator that also receives the layout.
• Leverage perceptual losses to stabilize GAN training and improve object shapes; enable interactive control by modifying the predicted layout.

Experimental results

Research questions

• RQ1 Can explicitly inferring a semantic layout from text improve the quality and interpretability of text-to-image synthesis on complex scenes?
• RQ2 Does conditioning image generation on an inferred layout yield images that better align with the input text than direct text-to-image methods?
• RQ3 To what extent do bounding box and mask predictions contribute to image fidelity and semantic consistency?
• RQ4 Can the inferred layout support controllable generation and automatic annotation of generated content?

Key findings

• The proposed method substantially improves Inception score over prior GAN-based text-to-image approaches on MS-COCO.
• Caption-based evaluation shows generated images have higher relevance to input text than baselines.
• Human evaluation indicates users more often rank images from the proposed method as best in relevance to text (about 60% of tasks).
• Ablations show that replacing predicted layouts with ground-truth layouts yields progressive gains, confirming layout quality impacts image generation.
• The model produces more recognizable and semantically meaningful images by leveraging explicit layout conditioning.