[Paper Review] Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis

Ceyuan Yang, Yujun Shen+1 more|arXiv (Cornell University)|Nov 21, 2019

Generative Adversarial Networks and Image SynthesisComputer Science60 references41 citations

TL;DR

The paper analyzes how layer-wise latent codes in StyleGAN and BigGAN induce a hierarchical, human-understandable semantic structure in scene synthesis, identifying layout, objects, attributes, and color scheme as emergent variation factors and showing how to manipulate them.

ABSTRACT

Despite the success of Generative Adversarial Networks (GANs) in image synthesis, there lacks enough understanding on what generative models have learned inside the deep generative representations and how photo-realistic images are able to be composed of the layer-wise stochasticity introduced in recent GANs. In this work, we show that highly-structured semantic hierarchy emerges as variation factors from synthesizing scenes from the generative representations in state-of-the-art GAN models, like StyleGAN and BigGAN. By probing the layer-wise representations with a broad set of semantics at different abstraction levels, we are able to quantify the causality between the activations and semantics occurring in the output image. Such a quantification identifies the human-understandable variation factors learned by GANs to compose scenes. The qualitative and quantitative results further suggest that the generative representations learned by the GANs with layer-wise latent codes are specialized to synthesize different hierarchical semantics: the early layers tend to determine the spatial layout and configuration, the middle layers control the categorical objects, and the later layers finally render the scene attributes as well as color scheme. Identifying such a set of manipulatable latent variation factors facilitates semantic scene manipulation.

Motivation & Objective

• Investigate what semantic factors GANs learn during scene synthesis across multiple abstraction levels (layout, objects, attributes, color).
• Quantify causality between layer-wise generator activations and output semantics in state-of-the-art GANs.
• Identify manipulatable latent variation factors and map them to generator layers to enable semantic scene editing.
• Demonstrate that hierarchical semantics emerge without external supervision and enable diverse scene manipulation.
• Show generalization of the approach to different GAN architectures (StyleGAN, BigGAN, PGGAN).

Proposed method

• Treat the GAN latent code as a layer-wise generative representation fed to multiple generator layers (layer-wise stochasticity).
• Define four abstraction levels (layout, objects, attributes, color) and use off-the-shelf classifiers to score semantics from synthesized images.
• Probe latent space by training linear SVM decision boundaries for each semantic concept treated as a binary task.
• Verify manipulatable variation factors by moving latent codes along boundary normals and re-scoring changes in semantics (Delta s_i).
• Perform independent, joint, and jittered manipulation to edit scenes across layers and semantics.
• Apply the approach to StyleGAN, BigGAN, and PGGAN across indoor/outdoor scenes; use FID/LSUN/Places data as described; quantify layer-wise specialization (layout bottom, color top).

Experimental results

Research questions

• RQ1 What semantic factors emerge in GANs when synthesizing scenes across multiple abstraction levels?
• RQ2 How are these semantic factors distributed across generator layers in StyleGAN/BigGAN/PGGAN?
• RQ3 Can we quantitatively identify and manipulate the emergent variation factors via layer-wise latent codes?
• RQ4 Do layer-wise latent representations generalize across different GAN architectures and scene categories?

Key findings

• A hierarchical semantic structure emerges in GAN representations: early layers control layout, middle layers control objects, and later layers render attributes and color schemes.
• Layer-wise latent codes enable manipulable scene editing by moving latent codes along semantic boundaries, producing diverse, semantically coherent edits.
• The middle layers encode category-specific objects, enabling category transformations (e.g., bedroom to living room) while preserving layout and high-level attributes.
• A re-scoring technique ranks semantically relevant variation factors by measuring semantic score changes when latent codes cross boundary directions.
• Experiments show consistent layer-to-semantics mapping across StyleGAN, BigGAN, and PGGAN, with quantitative validation via classifiers and a user study on layer relevance.
• Table 1 reports Fréchet Inception Distance (FID) values for multiple scene categories (e.g., bedroom 2.65; living room 5.16; kitchen 5.06; restaurant 4.03; bridge 6.42; church 4.82; tower 5.99; mixed 3.74).