[Paper Review] Object-driven Text-to-Image Synthesis via Adversarial Training

Wenbo Li, Pengchuan Zhang+5 more|arXiv (Cornell University)|Feb 27, 2019

Generative Adversarial Networks and Image Synthesis33 references46 citations

TL;DR

Obj-GAN introduces object-driven attentive generation and object-wise discrimination to synthesize complex scenes from text, improving COCO results over prior methods.

ABSTRACT

In this paper, we propose Object-driven Attentive Generative Adversarial Newtorks (Obj-GANs) that allow object-centered text-to-image synthesis for complex scenes. Following the two-step (layout-image) generation process, a novel object-driven attentive image generator is proposed to synthesize salient objects by paying attention to the most relevant words in the text description and the pre-generated semantic layout. In addition, a new Fast R-CNN based object-wise discriminator is proposed to provide rich object-wise discrimination signals on whether the synthesized object matches the text description and the pre-generated layout. The proposed Obj-GAN significantly outperforms the previous state of the art in various metrics on the large-scale COCO benchmark, increasing the Inception score by 27% and decreasing the FID score by 11%. A thorough comparison between the traditional grid attention and the new object-driven attention is provided through analyzing their mechanisms and visualizing their attention layers, showing insights of how the proposed model generates complex scenes in high quality.

Motivation & Objective

• Motivation to generate high-quality, semantically meaningful images from complex text descriptions.
• Need to model objects and their relations explicitly to handle COCO-scale scenes.
• Develop an object-driven attention mechanism to utilize word-level and object-level information during generation.
• Propose an object-wise discriminator to provide object-level feedback aligned with the layout.

Proposed method

• Two-step generation: semantic layout (bounding boxes and shapes) followed by image synthesis.
• An object-driven attentive image generator that uses class labels to query word contexts for each object region.
• An object-wise Fast R-CNN based discriminator that evaluates each bounding box against text and layout.
• An object-driven attention mechanism computing word-context for each object and distributing context across object regions.
• Patch-wise discriminators provide unconditional and text-conditioned feedback; final-stage object-wise discriminator provides object-specific feedback.
• Loss combines GAN objectives with DAMSM cross-modal similarity loss for word-level text–image alignment.

Experimental results

Research questions

• RQ1 Can object-driven attention improve fine-grained, multi-object image synthesis from descriptive text?
• RQ2 Does incorporating object-wise discrimination and layout conditioning improve realism and layout fidelity on COCO-scale scenes?
• RQ3 How does object-driven attention compare to grid attention in guiding high-quality generation?
• RQ4 What is the impact of using ground-truth versus predicted layouts on synthesis quality?
• RQ5 Do the proposed components generalize to novel, rare scene descriptions?

Key findings

• Obj-GAN substantially improves COCO text-to-image synthesis performance over prior methods.
• Using object-driven attention yields higher Inception scores and lower FID than grid attention variants.
• Obj-GAN with ground-truth layouts achieves the best qualitative and quantitative results, illustrating layout quality’s impact.
• Ablation shows object-driven attention outperforms grid attention in generating coherent foreground and background textures.
• Obj-GAN-SOTA variants achieve the best reported metrics on the COCO benchmark in the study.
• Qualitative results demonstrate better object fidelity and scene coherence, including novel descriptions.