[Paper Review] Improved StyleGAN Embedding: Where are the Good Latents?

Peihao Zhu, Rameen Abdal+3 more|arXiv (Cornell University)|Dec 13, 2020

Generative Adversarial Networks and Image Synthesis24 references59 citations

TL;DR

The paper analyzes where good StyleGAN latents reside using a new PN space, and proposes PN+-based regularization to improve inverted embeddings, achieving better reconstruction-editing trade-offs compared with state-of-the-art methods.

ABSTRACT

StyleGAN is able to produce photorealistic images that are almost indistinguishable from real photos. The reverse problem of finding an embedding for a given image poses a challenge. Embeddings that reconstruct an image well are not always robust to editing operations. In this paper, we address the problem of finding an embedding that both reconstructs images and also supports image editing tasks. First, we introduce a new normalized space to analyze the diversity and the quality of the reconstructed latent codes. This space can help answer the question of where good latent codes are located in latent space. Second, we propose an improved embedding algorithm using a novel regularization method based on our analysis. Finally, we analyze the quality of different embedding algorithms. We compare our results with the current state-of-the-art methods and achieve a better trade-off between reconstruction quality and editing quality.

Motivation & Objective

• Understand the distribution and geometry of latent codes in StyleGAN spaces to identify where good latents lie.
• Propose a regularized embedding method that improves reconstruction while preserving editability.
• Extend the analysis to high-capacity W+ space for better inversion.
• Provide comprehensive, quantitative and user-driven evaluation against prior methods.

Proposed method

• Introduce PN space as a whitened (P_N) latent space where latent distributions are more isotropic and easier to model.
• Extend to PN+ space for the W+ (18x512) latent representation used by StyleGAN2/I2S framework (unpacks 18 w_i codes).
• Propose a density-based regularizer in PN+ space via Mahalanobis-distance (d_M) to bias embeddings toward high-density regions without sacrificing reconstruction.
• Train embeddings with a combined loss L = LPIPS + L2 plus an additional regularization term lambda * ||v||^2 in PN+ space where v is the PN+-transformed latent code.
• Use LPIPS perceptual loss and downsampling strategies (bicubic in generator space, Lanczos for reference) for reconstruction quality.
• Evaluate II2S (the proposed method) against multiple baselines (I2S, PULSE, StyleGAN2, StyleGAN2Encoder, pSp) on reconstruction, editing (pose, age, lighting), and conditional embedding tasks.

Experimental results

Research questions

• RQ1 Where are the high-quality, usable latent codes located in StyleGAN latent spaces?
• RQ2 Does regularizing embeddings in PN/PN+ space yield better reconstruction while preserving editability compared to existing methods?
• RQ3 How does the proposed embedding perform across editing tasks (pose, age, lighting) and conditional tasks (colorization, inpainting, super-resolution)?
• RQ4 What is the impact of the regularizer strength lambda on reconstruction vs editing quality, according to human studies and perceptual metrics?

Key findings

• The PN space reveals that latent code quality correlates with distance from the origin, and the PN distance corresponds to a Mahalanobis distance.
• A PN+-based regularizer biases embeddings toward high-density regions of latent space, improving editability with negligible loss in reconstruction.
• II2S achieves a favorable trade-off between reconstruction and editing quality versus state-of-the-art baselines across multiple metrics and tasks.
• User studies favor II2S over competing methods for editing tasks and several conditional embedding tasks, with I2S sometimes better on reconstruction alone.
• In conditional embedding tasks (colorization, inpainting, super-resolution), II2S often outperforms competitors; some tasks still favor ground truth or other baselines depending on the metric.