[Paper Review] Semantic Image Synthesis with Spatially-Adaptive Normalization

Taesung Park, Ming-Yu Liu+2 more|arXiv (Cornell University)|Mar 18, 2019

Generative Adversarial Networks and Image Synthesis57 references261 citations

TL;DR

Introduces SPADE, a spatially-adaptive normalization layer that modulates activations with input semantic layouts, preserving semantic information and enabling high-fidelity, layout-consistent image synthesis with multi-modal and style-guided control.

ABSTRACT

We propose spatially-adaptive normalization, a simple but effective layer for synthesizing photorealistic images given an input semantic layout. Previous methods directly feed the semantic layout as input to the deep network, which is then processed through stacks of convolution, normalization, and nonlinearity layers. We show that this is suboptimal as the normalization layers tend to ``wash away'' semantic information. To address the issue, we propose using the input layout for modulating the activations in normalization layers through a spatially-adaptive, learned transformation. Experiments on several challenging datasets demonstrate the advantage of the proposed method over existing approaches, regarding both visual fidelity and alignment with input layouts. Finally, our model allows user control over both semantic and style. Code is available at https://github.com/NVlabs/SPADE .

Motivation & Objective

• Motivate semantic information preservation during normalization in conditional image synthesis.
• Develop a normalization layer that uses semantic layouts to modulate activations spatially.
• Build a compact generator without heavy encoder-decoder upfront, leveraging SPADE for semantic propagation.
• Demonstrate multi-modal and style-guided image synthesis controlled by noise or style images.
• Evaluate on challenging datasets to show improvements over state-of-the-art methods.

Proposed method

• Define SPADE: a learned, spatially-varying affine transformation (gamma, beta) conditioned on semantic masks m.
• Replace traditional normalization after convolutions with SPADE to preserve input semantics across layers.
• Use a lightweight generator with ResNet blocks and upsampling, where all normalization layers are SPADE-modulated.
• Train with a multi-scale discriminator and a pix2pixHD-like loss, using hinge loss instead of least squares.
• Enable multi-modal synthesis by feeding a random vector as input, allowing diverse outputs for the same semantic layout.
• Optionally incorporate an image encoder to guide style and enable style-controlled generation.

Experimental results

Research questions

• RQ1 Does spatially-adaptive modulation better preserve semantic information than unconditional or globally conditioned normalization?
• RQ2 Can SPADE-based generators achieve higher visual fidelity and semantic alignment across diverse datasets (COCO-Stuff, ADE20K, Cityscapes) than prior methods?
• RQ3 Does SPADE support multi-modal and style-guided synthesis without sacrificing realism or semantic accuracy?
• RQ4 How does the proposed approach perform relative to strong baselines like pix2pixHD, CRN, and SIMS on standard metrics?
• RQ5 What architectural and ablative choices (kernel size, input type, normalization variants) affect SPADE performance?

Key findings

• SPADE-based generator markedly improves mIoU and pixel accuracy over prior methods across COCO-Stuff, ADE20K, ADE20K-outdoor, and Cityscapes.
• On COCO-Stuff, mIoU improves to 37.4 (from 23.7 CRN) with FID 22.6, indicating higher fidelity and better semantic alignment.
• On ADE20K, mIoU improves to 38.5 (from 22.4 CRN) with FID 33.9, showing strong gains in outdoor and complex scenes.
• On Cityscapes, mIoU reaches 62.3 with FID 71.8, outperforming competing methods.
• Qualitative results show more realistic textures and fewer artifacts, with strong user preference for SPADE-produced images across datasets.
• The framework supports multi-modal outputs and style-guided synthesis while maintaining layout fidelity.