[论文解读] GAN Dissection: Visualizing and Understanding Generative Adversarial Networks

David Bau, Jun-Yan Zhu+5 more|arXiv (Cornell University)|Nov 26, 2018

Cellular Automata and Applications被引用 217

一句话总结

本论文提出一个框架，通过发现可解释的单位、通过干预测试它们的因果作用，并将其应用于调试和操控生成场景，以可视化和理解 GAN。

ABSTRACT

Generative Adversarial Networks (GANs) have recently achieved impressive results for many real-world applications, and many GAN variants have emerged with improvements in sample quality and training stability. However, they have not been well visualized or understood. How does a GAN represent our visual world internally? What causes the artifacts in GAN results? How do architectural choices affect GAN learning? Answering such questions could enable us to develop new insights and better models. In this work, we present an analytic framework to visualize and understand GANs at the unit-, object-, and scene-level. We first identify a group of interpretable units that are closely related to object concepts using a segmentation-based network dissection method. Then, we quantify the causal effect of interpretable units by measuring the ability of interventions to control objects in the output. We examine the contextual relationship between these units and their surroundings by inserting the discovered object concepts into new images. We show several practical applications enabled by our framework, from comparing internal representations across different layers, models, and datasets, to improving GANs by locating and removing artifact-causing units, to interactively manipulating objects in a scene. We provide open source interpretation tools to help researchers and practitioners better understand their GAN models.

研究动机与目标

• 在 GAN 生成器中识别与对象概念（例如树木、桌子）对应的单位。
• 量化所选单位对生成图像中对象存在与否的因果影响。
• 探索对象概念如何在场景中的上下文和背景下互动。
• 实现实际用途，如比较模型之间的表示、诊断伪影，以及交互式对象操控。

提出的方法

• Dissection: 测量单位激活与语义分割掩模之间的一致性以识别可解释单位（基于 IoU）。
• Intervention: 消融或插入一组单位，并利用分割差异计算对象存在的平均因果效应（ACE）。
• Optimize a continuous intervention vector to efficiently select a subset of units that maximizes ACE (with L2 regularization).
• Compare representations across layers, models, and datasets to understand how architecture and training affect learned objects.
• Use artifact identification to ablate units and improve image quality (FID and human judgments).
• Provide open-source tools and demos for interactive exploration.

实验结果

研究问题

• RQ1 GAN 是否在内部单位中学习了对象概念的显式、解耦表示？
• RQ2 是否可以通过干预特定单位来因果性地操纵生成图像中的对象存在？
• RQ3 层深、模型变体和训练数据如何影响可解释单位的涌现？
• RQ4 是否能够识别并移除导致伪影的单位以提升真实感？
• RQ5 上下文和环境如何影响将对象概念插入场景的效果？

主要发现

Metric / Condition	Original images	Artifacts ablated (ours)	Random units ablated
Fréchet Inception Distance (FID)	43.16	27.14	43.17
Human preference score (%)	—	72.4%	49.9%

• 某些单位在多样化外观中表现为对象检测器（如桌子、沙发），与分割映射在 IoU 方面具有有意义的匹配。
• 中后期层往往包含对应于对象及对象部件的可解释单位，而早期层编码低级特征。
• 架构选择（如小批量 stddev、逐像素归一化）会影响可解释单位的数量和多样性。
• 消融导致伪影的单位显著影响图像质量（FID 降低、人工偏好提高），相较随机消融效果更好。
• 干预表明插入或移除对象单位会产生上下文相关的结果，揭示 GAN 如何编码对象-上下文关系。
• 对一些场景，通过无效化较小、定向的单位集可以去除特定对象（如窗户、帘子），而其他对象（如桌子）较难消除。