[Paper Review] Dual Pyramid Generative Adversarial Networks for Semantic Image Synthesis

Shijie Li, Ming‐Ming Cheng+1 more|arXiv (Cornell University)|Oct 8, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes Dual Pyramid GAN (DP-GAN), a novel generative adversarial network for semantic image synthesis that uses a dual-encoder architecture to adaptively condition normalization blocks across multiple scales, enabling better generation of small and large objects. By unifying multi-scale supervision through feature matching and adversarial losses, DP-GAN achieves state-of-the-art performance, reducing FID by 3.6 and improving mIoU by 4.7 compared to prior methods.

ABSTRACT

The goal of semantic image synthesis is to generate photo-realistic images from semantic label maps. It is highly relevant for tasks like content generation and image editing. Current state-of-the-art approaches, however, still struggle to generate realistic objects in images at various scales. In particular, small objects tend to fade away and large objects are often generated as collages of patches. In order to address this issue, we propose a Dual Pyramid Generative Adversarial Network (DP-GAN) that learns the conditioning of spatially-adaptive normalization blocks at all scales jointly, such that scale information is bi-directionally used, and it unifies supervision at different scales. Our qualitative and quantitative results show that the proposed approach generates images where small and large objects look more realistic compared to images generated by state-of-the-art methods.

Motivation & Objective

• To address the challenge of generating photo-realistic images with consistent, realistic small and large objects from semantic label maps.
• To overcome limitations in existing GANs that produce patchy or faded objects due to poor scale-aware conditioning.
• To unify multi-scale supervision in both generator and discriminator for improved alignment with input layouts.
• To enhance object-level realism, especially for thin or small structures like traffic signs and bollards.
• To improve generalization and diversity in image generation while maintaining precise semantic alignment.

Proposed method

• Introduces a dual pyramid generator that learns scale-adaptive conditioning for normalization blocks using two parallel feature pyramids—one for semantic context and one for spatial adaptation.
• Employs a segmentation-based discriminator with multi-scale supervision via a multi-scale adversarial loss ($\mathcal{L}_{ms}$) and a multi-scale feature matching loss ($\mathcal{L}_{fm}$).
• Uses a dual-path architecture where the generator's normalization layers are conditioned on both the original label map and a scale-aware feature pyramid, enabling better scale generalization.
• Applies the feature matching loss ($\mathcal{L}_{fm}$) specifically to the decoder branch of the discriminator to improve feature-level alignment with real images.
• Unifies supervision across scales by combining pixel-wise segmentation supervision with adversarial and feature matching losses at multiple resolutions.
• Enables diverse outputs by using random noise vectors $z$ as input, allowing multiple realistic samples per label map.

Experimental results

Research questions

• RQ1 Can a dual pyramid architecture improve the generation of small and large objects in semantic image synthesis by adapting normalization conditioning to object scale?
• RQ2 Does multi-scale supervision in both generator and discriminator lead to better alignment with input semantic layouts and improved realism?
• RQ3 How does the placement of feature matching loss ($\mathcal{L}_{fm}$) affect performance in a segmentation-based GAN?
• RQ4 Can a unified multi-scale training framework outperform existing SOTA methods in both FID and mIoU metrics?
• RQ5 To what extent does the proposed method reduce artifacts like patchy or faded objects compared to SPADE and OASIS?

Key findings

• DP-GAN achieves a FID of 44.1, representing a 3.6-point improvement over OASIS (47.7), indicating significantly better image quality.
• The mIoU score reaches 73.6, a 4.7-point increase over OASIS (69.3), demonstrating superior semantic alignment with ground truth.
• The object-level mIoU improves by 5.9 points (from 61.5 to 67.4), showing marked gains in object-level realism and detail.
• Adding the multi-scale adversarial loss ($\mathcal{L}_{ms}$) to the encoder branch of the discriminator yields the best performance, improving FID and mIoU.
• The feature matching loss ($\mathcal{L}_{fm}$) improves mIoU by 3.7 points when applied to the decoder, but degrades performance when applied to the encoder.
• Ablation studies confirm that the combination of the proposed generator and discriminator is essential, as using them in isolation yields suboptimal results.