[Paper Review] Using latent space regression to analyze and leverage compositionality in GANs

Lucy Chai, Jonas Wulff+1 more|arXiv (Cornell University)|Mar 18, 2021

Generative Adversarial Networks and Image SynthesisComputer Science51 references30 citations

TL;DR

The paper trains a latent space regressor to map images into a fixed GAN’s latent code, enabling real-time, label-free editing and composition by leveraging the GAN’s latent priors to produce coherent composites from image parts.

ABSTRACT

In recent years, Generative Adversarial Networks have become ubiquitous in both research and public perception, but how GANs convert an unstructured latent code to a high quality output is still an open question. In this work, we investigate regression into the latent space as a probe to understand the compositional properties of GANs. We find that combining the regressor and a pretrained generator provides a strong image prior, allowing us to create composite images from a collage of random image parts at inference time while maintaining global consistency. To compare compositional properties across different generators, we measure the trade-offs between reconstruction of the unrealistic input and image quality of the regenerated samples. We find that the regression approach enables more localized editing of individual image parts compared to direct editing in the latent space, and we conduct experiments to quantify this independence effect. Our method is agnostic to the semantics of edits, and does not require labels or predefined concepts during training. Beyond image composition, our method extends to a number of related applications, such as image inpainting or example-based image editing, which we demonstrate on several GANs and datasets, and because it uses only a single forward pass, it can operate in real-time. Code is available on our project page: https://chail.github.io/latent-composition/.

Motivation & Objective

• Investigate whether a pretrained GAN’s latent space contains compositional representations of objects and scenes.
• Develop a fast, feed-forward latent regressor to map images to GAN latent codes with missing-pixel handling.
• Demonstrate real-time image editing and collage composition by projecting unrealistic inputs onto the GAN manifold.
• Quantify the tradeoffs between input reconstruction fidelity and output realism across architectures and datasets.
• Show applications of the approach to inpainting, scene completion, and dataset rebalancing without label supervision.

Proposed method

• Train a latent regression network E to predict the latent code z from an input image x, using a loss that includes image reconstruction, perceptual, and latent-code reconstruction terms.
• Fix the GAN generator G (e.g., ProGAN, StyleGAN) while training E, enabling forward-pass inversion without updating G.
• Handle missing data by providing a masked input (x_m, m) and augmenting the loss to learn consistent latent codes despite unknown pixels.
• Use the regressor-G combination as a strong image prior to project collages of image parts into the image manifold and generate coherent composites.
• Formulate image composition by creating collages x_clg from parts, then reprojecting via E and G to obtain x_rec, enabling blending, alignment, and inpainting.
• Compare reconstruction-realism tradeoffs across encoder-based, optimization-based, and autoencoder approaches using metrics like masked L1 and FID across multiple datasets.

Experimental results

Research questions

• RQ1 Does the pretrained GAN latent code encode compositional structure that allows independent editing of image parts?
• RQ2 Can a single forward-pass latent regressor enable realistic composition of image parts without labeled attributes?
• RQ3 How does composition via latent regression differ from simple latent or pixel-space interpolation?
• RQ4 To what extent can the regressor handle incomplete/missing input regions and still produce coherent outputs?
• RQ5 How do compositional properties vary across different GAN architectures and datasets?

Key findings

• A regressor paired with a fixed generator provides a strong image prior that can render realistic composites from collage inputs.
• The generator’s latent code already encodes compositional structure, enabling localized edits without labeled attributes.
• Edit operations can be performed in real-time with a single forward pass, avoiding iterative optimization.
• Masking inputs to simulate missing data allows the model to complete scenes coherently while respecting context.
• Collage-based composition via latent regression balances input preservation with output realism, demonstrating a Pareto-like tradeoff between L1 reconstruction and FID realism.
• Across architectures, StyleGAN shows more localized variation control, while ProGAN provides different reconstruction-realism tradeoffs.