[Paper Review] Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis

Xihui Liu, Guojun Yin+3 more|arXiv (Cornell University)|Oct 15, 2019

Generative Adversarial Networks and Image Synthesis92 citations

TL;DR

Introduces CC-FPSE: a generator using layout-conditioned, depthwise separable convolutions and a feature pyramid semantics-embedding discriminator to synthesize high-fidelity, semantically aligned images from semantic layouts. Achieves state-of-the-art on Cityscapes, COCO-Stuff, and ADE20K.

ABSTRACT

Semantic image synthesis aims at generating photorealistic images from semantic layouts. Previous approaches with conditional generative adversarial networks (GAN) show state-of-the-art performance on this task, which either feed the semantic label maps as inputs to the generator, or use them to modulate the activations in normalization layers via affine transformations. We argue that convolutional kernels in the generator should be aware of the distinct semantic labels at different locations when generating images. In order to better exploit the semantic layout for the image generator, we propose to predict convolutional kernels conditioned on the semantic label map to generate the intermediate feature maps from the noise maps and eventually generate the images. Moreover, we propose a feature pyramid semantics-embedding discriminator, which is more effective in enhancing fine details and semantic alignments between the generated images and the input semantic layouts than previous multi-scale discriminators. We achieve state-of-the-art results on both quantitative metrics and subjective evaluation on various semantic segmentation datasets, demonstrating the effectiveness of our approach.

Motivation & Objective

• Motivate more expressive use of semantic layouts in the generator beyond affine modulation of activations.
• Propose layout-conditioned, spatially varying convolution kernels predicted from the semantic map to control generation at each location.
• Improve realism and semantic alignment with a feature pyramid semantics-embedding discriminator.
• Demonstrate state-of-the-art performance on multiple semantic segmentation datasets.
• Provide an efficient, scalable architecture leveraging depthwise separable convolutions to manage parameters.

Proposed method

• Propose conditional convolution blocks where spatially varying kernels are predicted from the semantic layout.
• Factorize convolutions into depthwise and pointwise components to reduce parameters.
• Use a global-context-aware, feature-p pyramid weight predictor to generate per-location convolution weights and attention maps.
• Introduce a feature pyramid semantics-embedding discriminator that enforces high-fidelity details and semantic alignment at multiple scales.
• Train with hinge loss adversarial objectives plus perceptual and style/feature-m matching losses.

Experimental results

Research questions

• RQ1 How can semantic layouts be used to condition convolutional kernels in the generator for more precise, location-specific synthesis?
• RQ2 Does predicting spatially varying depthwise kernels from layout improve image quality and semantic alignment over affine feature modulation methods?
• RQ3 Can a single, feature pyramid-based discriminator with semantics embedding better enforce detail fidelity and layout-consistency than multi-scale PatchGANs?
• RQ4 What are the quantitative gains on Cityscapes, COCO-Stuff, and ADE20K when using CC-FPSE compared to prior methods like SPADE and pix2pixHD?

Key findings

• The proposed CC-FPSE method achieves state-of-the-art results on Cityscapes, COCO-Stuff, and ADE20K across metrics (mIOU, accuracy, FID).
• Conditional convolution blocks with depthwise-separable parameterization enable per-location layout control with substantially fewer parameters than full dynamic filters.
• A global-context-aware weight prediction network with a feature pyramid improves weight predictions by incorporating long-range context.
• The feature pyramid semantics-embedding discriminator improves alignment between generated images and the input semantic layouts, yielding better texture/details and semantic consistency.
• Human perceptual evaluation shows higher preference for CC-FPSE over SPADE on Cityscapes, COCO-Stuff, and ADE20K.