[論文レビュー] GAN Dissection: Visualizing and Understanding Generative Adversarial Networks

David Bau, Jun-Yan Zhu+5 more|arXiv (Cornell University)|Nov 26, 2018

Cellular Automata and Applications被引用数 217

ひとこと要約

この論文は、解釈可能な単位を発見し介入で因果役割を検証することでGANを可視化・理解する枠組みを提案し、生成シーンのデバッグ・操作に適用する。

ABSTRACT

Generative Adversarial Networks (GANs) have recently achieved impressive results for many real-world applications, and many GAN variants have emerged with improvements in sample quality and training stability. However, they have not been well visualized or understood. How does a GAN represent our visual world internally? What causes the artifacts in GAN results? How do architectural choices affect GAN learning? Answering such questions could enable us to develop new insights and better models. In this work, we present an analytic framework to visualize and understand GANs at the unit-, object-, and scene-level. We first identify a group of interpretable units that are closely related to object concepts using a segmentation-based network dissection method. Then, we quantify the causal effect of interpretable units by measuring the ability of interventions to control objects in the output. We examine the contextual relationship between these units and their surroundings by inserting the discovered object concepts into new images. We show several practical applications enabled by our framework, from comparing internal representations across different layers, models, and datasets, to improving GANs by locating and removing artifact-causing units, to interactively manipulating objects in a scene. We provide open source interpretation tools to help researchers and practitioners better understand their GAN models.

研究の動機と目的

• GAN生成器内のオブジェクト概念に対応するユニットを同定する。
• 選択したユニットが生成画像におけるオブジェクトの有無に及ぼす因果的影響を定量化する。
• シーン内でのオブジェクト概念がコンテキストや背景とどのように相互作用するかを探る。
• モデル間の表現比較、アーティファクト診断、対話的なオブジェクト操作などの実用的用途を可能にする。

提案手法

• Dissection: ユニット活性化とセマンティックセグメンテーションマスクとの一致度を測定して解釈可能なユニットを特定する（IoUベース） 。
• Intervention: ユニットのセットを破壊または挿入し、セグメンテーションの差分を用いてオブジェクトの有無に対する平均因果効果（ACE）を計算する。
• Optimize a continuous intervention vector to efficiently select a subset of units that maximizes ACE (with L2 regularization).
• 層、モデル、データセットを横断して表現を比較し、アーキテクチャと訓練が学習されるオブジェクトにどう影響するかを理解する。
• Use artifact identification to ablate units and improve image quality (FID and human judgments).
• Provide open-source tools and demos for interactive exploration.

実験結果

リサーチクエスチョン

• RQ1 Do GANs learn explicit, disentangled representations of object concepts within internal units?
• RQ2 Can we causally manipulate object presence in generated images by intervening on specific units?
• RQ3 How do layer depth, model variant, and training data affect the emergence of interpretable units?
• RQ4 Can we identify and remove artifact-causing units to improve realism?
• RQ5 How do context and surroundings influence the effect of inserting object concepts into scenes?

主な発見

Metric / Condition	Original images	Artifacts ablated (ours)	Random units ablated
Fréchet Inception Distance (FID)	43.16	27.14	43.17
Human preference score (%)	—	72.4%	49.9%

• Some units emerge as object detectors (e.g., tables, sofas) with meaningful IoU matches to segmentation maps across diverse appearances.
• Mid-to-late layers tend to host interpretable units corresponding to objects and object parts, while early layers encode low-level features.
• Architectural choices (e.g., minibatch stddev, pixelwise normalization) affect the number and variety of interpretable units.
• Ablating artifact-causing units significantly improves image quality (lower FID and higher human preference) compared to random ablations.
• Interventions show that inserting or removing object units yields context-dependent results, revealing how GANs encode object-context relationships.
• Nullifying small, targeted sets of units can remove specific objects (e.g., windows, curtains) in some scenes, while others (e.g., tables) are harder to eliminate.