[Paper Review] Semantic Image Synthesis via Diffusion Models

Wengang Zhou, Weilun Wang+5 more|arXiv (Cornell University)|Jun 30, 2022

Generative Adversarial Networks and Image SynthesisComputer Science87 citations

TL;DR

The paper introduces Semantic Diffusion Model (SDM), a DDPM-based framework that separately processes semantic layouts and noisy images with SPADE-like conditioning, achieving high fidelity and diversity in semantic image synthesis with classifier-free guidance.

ABSTRACT

Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks compared with Generative Adversarial Nets (GANs). Recent work on semantic image synthesis mainly follows the de facto GAN-based approaches, which may lead to unsatisfactory quality or diversity of generated images. In this paper, we propose a novel framework based on DDPM for semantic image synthesis. Unlike previous conditional diffusion model directly feeds the semantic layout and noisy image as input to a U-Net structure, which may not fully leverage the information in the input semantic mask, our framework processes semantic layout and noisy image differently. It feeds noisy image to the encoder of the U-Net structure while the semantic layout to the decoder by multi-layer spatially-adaptive normalization operators. To further improve the generation quality and semantic interpretability in semantic image synthesis, we introduce the classifier-free guidance sampling strategy, which acknowledge the scores of an unconditional model for sampling process. Extensive experiments on four benchmark datasets demonstrate the effectiveness of our proposed method, achieving state-of-the-art performance in terms of fidelity (FID) and diversity (LPIPS). Our code and pretrained models are available at https://github.com/WeilunWang/semantic-diffusion-model.

Motivation & Objective

• Develop a diffusion-model framework for semantic image synthesis that surpasses GAN-based methods in fidelity and diversity.
• Leverage separate processing of semantic masks and noisy inputs to better utilize semantic information.
• Improve sampling quality and semantic correspondence via classifier-free guidance.
• Demonstrate strong performance on Cityscapes, ADE20K, CelebAMask-HQ, and COCO-Stuff datasets.

Proposed method

• Use a conditional denoising diffusion network (SDM) where the noisy image goes through the encoder while the semantic layout is injected into the decoder via multi-layer spatially-adaptive normalization (SPADE).
• Adopt SDEResblocks in the encoder with attention and timestep-aware scaling for denoising.
• Inject semantic information into the decoder with SPADE-like conditioning (SDDResblock) to guide denoising.
• Train with a denoising loss plus a variances prediction loss to maximize likelihood (L_simple + lambda * L_vlb).
• Apply classifier-free guidance by mixing conditional and unconditional predictions during sampling to boost fidelity and semantic alignment (epsilon_theta(y_t|x) + s*(epsilon_theta(y_t|x) - epsilon_theta(y_t|empty)).
• Optionally perform multimodal, diverse generation by leveraging the stochastic diffusion process.

Experimental results

Research questions

• RQ1 Can a diffusion-based framework outperform GAN-based methods in fidelity and diversity for semantic image synthesis?
• RQ2 Does separating the conditioning information (semantic mask) from the noisy image improve semantic relevance and visual quality?
• RQ3 What is the impact of classifier-free guidance on fidelity and semantic alignment in conditional diffusion models?
• RQ4 How does SDM perform on four benchmark datasets in terms of FID, LPIPS, and mIoU-based semantic interpretability?

Key findings

• SDM achieves state-of-the-art FID and LPIPS on four benchmarks compared to prior methods.
• Embedding semantic layouts via multi-layer SPADE-like conditioning in the decoder markedly improves fidelity and semantic relevance over simple concatenation.
• Classifier-free guidance substantially improves mIoU and FID with a modest change in LPIPS, yielding better semantic alignment.
• SDM provides high-quality, diverse semantic image synthesis, including multimodal generation and capable semantic editing in real images.