[Paper Review] DTGAN: Dual Attention Generative Adversarial Networks for Text-to-Image Generation

Zhenxing Zhang, Lambert Schomaker|arXiv (Cornell University)|Nov 5, 2020

Generative Adversarial Networks and Image Synthesis55 references38 citations

TL;DR

DTGAN proposes a single-generator/discriminator framework for text-to-image generation using dual attention modules—channel-aware and pixel-aware attention—to enhance semantic consistency and image quality. By integrating Conditional Adaptive Instance-Layer Normalization (CAdaILN) and a novel visual loss, DTGAN achieves state-of-the-art FID scores on CUB and COCO datasets, outperforming multi-stage models with improved training stability and resolution fidelity.

ABSTRACT

Most existing text-to-image generation methods adopt a multi-stage modular architecture which has three significant problems: 1) Training multiple networks increases the run time and affects the convergence and stability of the generative model; 2) These approaches ignore the quality of early-stage generator images; 3) Many discriminators need to be trained. To this end, we propose the Dual Attention Generative Adversarial Network (DTGAN) which can synthesize high-quality and semantically consistent images only employing a single generator/discriminator pair. The proposed model introduces channel-aware and pixel-aware attention modules that can guide the generator to focus on text-relevant channels and pixels based on the global sentence vector and to fine-tune original feature maps using attention weights. Also, Conditional Adaptive Instance-Layer Normalization (CAdaILN) is presented to help our attention modules flexibly control the amount of change in shape and texture by the input natural-language description. Furthermore, a new type of visual loss is utilized to enhance the image resolution by ensuring vivid shape and perceptually uniform color distributions of generated images. Experimental results on benchmark datasets demonstrate the superiority of our proposed method compared to the state-of-the-art models with a multi-stage framework. Visualization of the attention maps shows that the channel-aware attention module is able to localize the discriminative regions, while the pixel-aware attention module has the ability to capture the globally visual contents for the generation of an image.

Motivation & Objective

• To address the instability, high computational cost, and poor early-stage image quality in multi-stage text-to-image GANs.
• To improve semantic consistency between text descriptions and generated images without relying on multiple generators or discriminators.
• To enhance image quality through a new visual loss that preserves shape and color uniformity.
• To enable fine-grained control over texture and shape changes using conditional normalization.

Proposed method

• Introduces a channel-aware attention module that computes attention weights between the global sentence vector and feature maps to highlight text-relevant channels.
• Employs a pixel-aware attention module that generates spatial attention maps to guide the generator toward globally salient image regions.
• Proposes Conditional Adaptive Instance-Layer Normalization (CAdaILN), which adaptively blends instance and layer normalization using the sentence vector to control feature modulation.
• Designs a novel visual loss based on L1 distance between deep features of generated and real images to improve perceptual quality and color distribution.
• Uses a unified generator/discriminator pair to replace multi-stage architectures, reducing training complexity and improving convergence.
• Applies attention maps to directly fine-tune feature maps rather than replacing them, enabling more precise feature refinement.

Experimental results

Research questions

• RQ1 Can a single generator/discriminator pair achieve state-of-the-art performance in text-to-image generation without multi-stage training?
• RQ2 How can attention mechanisms be designed to improve both semantic alignment and image detail quality?
• RQ3 What is the impact of a new visual loss on shape and color fidelity in generated images?
• RQ4 How does CAdaILN compare to standard normalization layers in controlling texture and shape modulation?
• RQ5 To what extent do dual attention modules enhance feature refinement across different image scales?

Key findings

• DTGAN achieves a Fréchet Inception Distance (FID) of 16.35 on the CUB dataset, outperforming existing state-of-the-art models.
• The model achieves an Inception Score (IS) of 4.88 on CUB, demonstrating high-quality and diverse image generation.
• Ablation studies confirm that CAdaILN with sentence-level conditioning improves IS by 0.17 and reduces FID by 3.27 compared to batch normalization baselines.
• The visual loss module significantly improves image quality, as shown by more vivid shapes and perceptually uniform color distributions in qualitative comparisons.
• Attention map visualizations confirm that channel-aware attention localizes discriminative parts (e.g., beaks, eyes), while pixel-aware attention captures global image structure.
• The model without the visual loss fails to generate long-wingspan birds with realistic shapes and suffers from poor color detail, confirming the visual loss’s critical role.