[Paper Review] Dual Attention GANs for Semantic Image Synthesis

Hao Tang, Song Bai+1 more|arXiv (Cornell University)|Aug 29, 2020

Generative Adversarial Networks and Image Synthesis48 references4 citations

TL;DR

This paper proposes Dual Attention GANs (DAGAN), a novel framework for semantic image synthesis that enhances photo-realistic image generation by integrating position-wise spatial attention and scale-wise channel attention modules. By modeling semantic consistency across spatial and channel dimensions without architectural changes or extra training, DAGAN achieves state-of-the-art performance with fewer parameters, significantly reducing blur and artifacts compared to prior methods.

ABSTRACT

In this paper, we focus on the semantic image synthesis task that aims at transferring semantic label maps to photo-realistic images. Existing methods lack effective semantic constraints to preserve the semantic information and ignore the structural correlations in both spatial and channel dimensions, leading to unsatisfactory blurry and artifact-prone results. To address these limitations, we propose a novel Dual Attention GAN (DAGAN) to synthesize photo-realistic and semantically-consistent images with fine details from the input layouts without imposing extra training overhead or modifying the network architectures of existing methods. We also propose two novel modules, i.e., position-wise Spatial Attention Module (SAM) and scale-wise Channel Attention Module (CAM), to capture semantic structure attention in spatial and channel dimensions, respectively. Specifically, SAM selectively correlates the pixels at each position by a spatial attention map, leading to pixels with the same semantic label being related to each other regardless of their spatial distances. Meanwhile, CAM selectively emphasizes the scale-wise features at each channel by a channel attention map, which integrates associated features among all channel maps regardless of their scales. We finally sum the outputs of SAM and CAM to further improve feature representation. Extensive experiments on four challenging datasets show that DAGAN achieves remarkably better results than state-of-the-art methods, while using fewer model parameters. The source code and trained models are available at https://github.com/Ha0Tang/DAGAN.

Motivation & Objective

• To address the limitations of existing GAN-based methods in semantic image synthesis, which suffer from blurry and inconsistent outputs due to weak semantic constraints and ignored structural correlations.
• To improve intra-class semantic consistency by explicitly modeling spatial and channel-wise attention in feature representations.
• To design lightweight, general-purpose modules that can be seamlessly integrated into existing GAN architectures without modifying network structures or adding training overhead.
• To achieve superior image quality and semantic consistency across diverse datasets, including high-resolution and fine-grained object synthesis.

Proposed method

• Proposes a Position-Wise Spatial Attention Module (SAM) that computes attention maps to correlate pixels with the same semantic label regardless of spatial distance, enhancing local and global semantic coherence.
• Introduces a Scale-Wise Channel Attention Module (CAM) that adaptively recalibrates feature maps across different scales and channels, emphasizing relevant features for improved representation.
• Combines the outputs of SAM and CAM via element-wise addition to enrich feature representations and strengthen semantic consistency.
• Employs a standard GAN framework with a generator and discriminator, where the proposed attention modules are inserted into the generator without altering the backbone architecture.
• Uses a learnable attention mechanism that computes attention weights based on global context, enabling the model to focus on semantically relevant regions and features.
• Trains the entire DAGAN model end-to-end using adversarial loss and perceptual loss, ensuring realistic and semantically accurate image synthesis.

Experimental results

Research questions

• RQ1 Can modeling spatial and channel-wise attention jointly improve semantic consistency in generated images?
• RQ2 Does the proposed attention mechanism reduce blur and artifacts compared to existing GAN-based methods?
• RQ3 Can the attention modules be applied to existing GAN architectures without architectural changes or additional training cost?
• RQ4 How effective is the dual attention mechanism on diverse datasets with varying resolutions and object complexity?

Key findings

• On the Cityscapes dataset, DAGAN achieves an mIoU of 66.1, a 4.8-point improvement over the baseline (61.3), demonstrating strong semantic consistency.
• The FID score on Cityscapes is 60.3, a 11.5-point reduction from the baseline (71.8), indicating significantly improved image realism and diversity.
• Compared to the state-of-the-art GauGAN, DAGAN produces more accurate and consistent semantic maps, with highlighted improvements in object regions such as roads and vehicles.
• The ablation study confirms that combining both SAM and CAM-II yields the best performance, outperforming individual modules and prior methods like (Cai et al., 2019) by a large margin.
• The proposed modules are lightweight and generalizable, enabling integration into existing GAN frameworks with negligible computational overhead.