[论文解读] Semantic Image Synthesis with Spatially-Adaptive Normalization

Taesung Park, Ming-Yu Liu+2 more|arXiv (Cornell University)|Mar 18, 2019

Generative Adversarial Networks and Image Synthesis参考文献 57被引用 261

一句话总结

引入 SPADE，一种空间自适应归一化层，通过输入的语义布局调制激活，保留语义信息并实现高保真、与布局一致的图像合成，支持多模态与风格引导控制。

ABSTRACT

We propose spatially-adaptive normalization, a simple but effective layer for synthesizing photorealistic images given an input semantic layout. Previous methods directly feed the semantic layout as input to the deep network, which is then processed through stacks of convolution, normalization, and nonlinearity layers. We show that this is suboptimal as the normalization layers tend to ``wash away'' semantic information. To address the issue, we propose using the input layout for modulating the activations in normalization layers through a spatially-adaptive, learned transformation. Experiments on several challenging datasets demonstrate the advantage of the proposed method over existing approaches, regarding both visual fidelity and alignment with input layouts. Finally, our model allows user control over both semantic and style. Code is available at https://github.com/NVlabs/SPADE .

研究动机与目标

• 在条件图像合成中推动归一化阶段保留语义信息。
• 开发一种使用语义布局来对激活进行空间调制的归一化层。
• 构建一个紧凑的生成器，无需前置的大型编码器-解码器，利用 SPADE 进行语义传播。
• 展示由噪声或风格图像控制的多模态和风格引导图像合成。
• 在具有挑战性的数据集上评估，以显示相对于最先进方法的改进。

提出的方法

• 定义 SPADE：在语义掩码 m 条件下学习的、空间可变仿射变换（gamma, beta）。
• 将卷积后传统归一化替换为 SPADE，以在各层中保留输入语义。
• 使用具有 ResNet 块和上采样的轻量生成器，其中所有归一化层均为 SPADE 调制。
• 使用多尺度判别器和类似 pix2pixHD 的损失训练，采用 hinge 损失替代最小二乘。
• 通过输入随机向量来实现多模态合成，使同一语义布局产生多样化输出。
• 可选地加入图像编码器以引导风格并实现风格控制生成。

实验结果

研究问题

• RQ1 空间自适应调制是否比无条件或全局条件归一化更好地保留语义信息？
• RQ2 基于 SPADE 的生成器是否能在多样数据集（COCO-Stuff、ADE20K、Cityscapes）上实现比先前方法更高的视觉保真度和语义对齐？
• RQ3 SPADE 是否在不牺牲真实感或语义准确性的情况下，支持多模态和风格引导的合成？
• RQ4 在标准指标上，与强基线如 pix2pixHD、CRN 和 SIMS 相比，所提出的方法表现如何？
• RQ5 哪些架构和消融选择（卷积核大小、输入类型、归一化变体）会影响 SPADE 的性能？

主要发现

• SPADE-based generator markedly improves mIoU and pixel accuracy over prior methods across COCO-Stuff, ADE20K, ADE20K-outdoor, and Cityscapes.
• On COCO-Stuff, mIoU improves to 37.4 (from 23.7 CRN) with FID 22.6, indicating higher fidelity and better semantic alignment.
• On ADE20K, mIoU improves to 38.5 (from 22.4 CRN) with FID 33.9, showing strong gains in outdoor and complex scenes.
• On Cityscapes, mIoU reaches 62.3 with FID 71.8, outperforming competing methods.
• Qualitative results show more realistic textures and fewer artifacts, with strong user preference for SPADE-produced images across datasets.
• The framework supports multi-modal outputs and style-guided synthesis while maintaining layout fidelity.