[Paper Review] MirrorGAN: Learning Text-to-image Generation by Redescription

Tingting Qiao, Jing Zhang+2 more|arXiv (Cornell University)|Mar 14, 2019

Multimodal Machine Learning Applications42 references72 citations

TL;DR

MirrorGAN introduces a text-to-image-to-text framework (STEM, GLAM, STREAM) with global-local attention and a reconstruction loss to improve semantic consistency between text and generated images, achieving state-of-the-art results on CUB and COCO.

ABSTRACT

Generating an image from a given text description has two goals: visual realism and semantic consistency. Although significant progress has been made in generating high-quality and visually realistic images using generative adversarial networks, guaranteeing semantic consistency between the text description and visual content remains very challenging. In this paper, we address this problem by proposing a novel global-local attentive and semantic-preserving text-to-image-to-text framework called MirrorGAN. MirrorGAN exploits the idea of learning text-to-image generation by redescription and consists of three modules: a semantic text embedding module (STEM), a global-local collaborative attentive module for cascaded image generation (GLAM), and a semantic text regeneration and alignment module (STREAM). STEM generates word- and sentence-level embeddings. GLAM has a cascaded architecture for generating target images from coarse to fine scales, leveraging both local word attention and global sentence attention to progressively enhance the diversity and semantic consistency of the generated images. STREAM seeks to regenerate the text description from the generated image, which semantically aligns with the given text description. Thorough experiments on two public benchmark datasets demonstrate the superiority of MirrorGAN over other representative state-of-the-art methods.

Motivation & Objective

• Address the gap between visual realism and semantic consistency in T2I generation.
• Leverage a dual-regularization idea by enforcing image redescriptions to align with the input text.
• Propose a unified T2I–I2T framework to improve cross-modal semantic alignment.
• Introduce a global-local collaborative attention mechanism within cascaded generators.
• Regularize training with a CE-based text-semantics reconstruction loss.

Proposed method

• Introduce STEM to obtain word- and sentence-level text embeddings, with conditioning augmentation.
• Use GLAM as a cascaded multi-stage generator with global-local attention that combines Attn_w (word) and Attn_s (sentence) at each stage.
• Incorporate STREAM to regenerate a text description from the generated image and align semantics via a cross-entropy loss.
• Train with two adversarial losses: visual realism and text-image paired semantic consistency, plus a CE-based text-semantics reconstruction loss (L_stream).
• Optimize generators across stages with an objective that sums stage losses and L_stream, while discriminators learn to distinguish real/fake images and semantic consistency.

Experimental results

Research questions

• RQ1 Can a text-to-image model benefit from enforcing a reverse text generation (I2T) to ensure semantic alignment with the input description?
• RQ2 Does a global-local attentive cascade improve semantic consistency and diversity beyond word-level attention alone?
• RQ3 Does a cross-modal reconstruction loss improve semantic alignment without sacrificing image realism?
• RQ4 What is the impact of integrating STREAM with GLAM on standard T2I benchmarks?
• RQ5 How does MirrorGAN perform on CUB and COCO compared to state-of-the-art methods in terms of Inception Score and R-precision?

Key findings

• MirrorGAN achieves higher Inception Scores than state-of-the-art methods on both CUB and COCO (CUB: 4.56; COCO: 26.47).
• MirrorGAN outperforms AttnGAN on R-precision across top-1, top-2, and top-3 settings on both datasets (CUB: 57.67/58.52/60.42; COCO: 74.52/76.87/80.21).
• Incorporating STREAM and GLAM yields significant gains over a word-attention baseline and over prior methods, establishing new state-of-the-art results.
• Ablation shows increasing the loss weight lambda improves both Inception Score and R-precision, with lambda=20 providing strong performance on COCO (Inception 26.47; R-precision 74.52).
• Qualitative and human perceptual studies indicate MirrorGAN produces more semantically consistent and authentic images than AttnGAN, with better alignment to input descriptions.