[Paper Review] Interpreting the Latent Space of GANs for Semantic Face Editing

Yujun Shen, Jinjin Gu+2 more|arXiv (Cornell University)|Jul 25, 2019

Generative Adversarial Networks and Image Synthesis59 references75 citations

TL;DR

The paper introduces InterFaceGAN, a framework that discovers and exploits linear latent subspaces corresponding to facial attributes in GANs to enable semantic face editing without retraining. It demonstrates disentangled and controllable attribute manipulation across PGGAN, StyleGAN, and real images via GAN inversion and projection-based conditioning.

ABSTRACT

Despite the recent advance of Generative Adversarial Networks (GANs) in high-fidelity image synthesis, there lacks enough understanding of how GANs are able to map a latent code sampled from a random distribution to a photo-realistic image. Previous work assumes the latent space learned by GANs follows a distributed representation but observes the vector arithmetic phenomenon. In this work, we propose a novel framework, called InterFaceGAN, for semantic face editing by interpreting the latent semantics learned by GANs. In this framework, we conduct a detailed study on how different semantics are encoded in the latent space of GANs for face synthesis. We find that the latent code of well-trained generative models actually learns a disentangled representation after linear transformations. We explore the disentanglement between various semantics and manage to decouple some entangled semantics with subspace projection, leading to more precise control of facial attributes. Besides manipulating gender, age, expression, and the presence of eyeglasses, we can even vary the face pose as well as fix the artifacts accidentally generated by GAN models. The proposed method is further applied to achieve real image manipulation when combined with GAN inversion methods or some encoder-involved models. Extensive results suggest that learning to synthesize faces spontaneously brings a disentangled and controllable facial attribute representation.

Motivation & Objective

• Identify semantic attributes encoded in GAN latent spaces for face synthesis.
• Demonstrate that linear hyperplanes in latent space separate binary attributes (pose, smile, age, gender, eyeglasses).
• Show that attribute representations become disentangled after linear transformations and can be manipulated with minimal entanglement.
• Enable precise attribute editing on fixed GAN models without retraining, including real image editing via inversion.
• Extend the approach to conditional manipulation to decorrelate attributes and correct artifacts in generated faces.

Proposed method

• Model the semantic score of an image as a linear function of the latent code: s = f(g(z)) ≈ λ n^T z for a hyperplane with normal n.
• Train linear SVM boundaries in latent space for binary attributes (pose, smile, age, gender, eyeglasses) to identify attribute hyperplanes.
• Use latent code edits z_edit = z + α n to manipulate single attributes; α controls strength of the semantic change.
• Apply conditional manipulation by orthogonalizing attribute directions via projection: n1 -> n1 - (n1^T n2) n2 to decouple attributes.
• Extend to real-image editing via GAN inversion (optimization-based or encoder-based) to map real images to latent codes before manipulation.
• Compare latent spaces of PGGAN and StyleGAN, and analyze Z vs W spaces in StyleGAN, including the use of projection-based decorrelation in latent space.

Experimental results

Research questions

• RQ1 Can binary facial attributes be separated by linear hyperplanes in GAN latent space?
• RQ2 Do linear latent subspaces correspond to distinct semantic attributes and can they be manipulated independently?
• RQ3 Can conditional projections decorrelate attributes to enable multi-attribute editing without unintended changes?
• RQ4 How well do these semantics transfer to real images via GAN inversion or encoder-based models?
• RQ5 What differences arise between latent spaces in PGGAN and StyleGAN regarding disentanglement and editability?

Key findings

• Linear hyperplanes in latent space separate attributes with high validation accuracy (pose 100.0%, smile 96.9%, age 97.9%, gender 98.7%, eyeglasses 95.6%; all attributes 90.3% for pose on all data, 78.5% for smile, 75.3% for age, 84.2% for gender, 80.1% for eyeglasses].
• Semantics encoded in GANs can be manipulated by moving along the corresponding latent directions, enabling changes in gender, age, expression, eyeglasses, pose, and artifact correction.
• Attributes show varying degrees of entanglement; conditional manipulation via projection reduces unintended changes in other attributes (e.g., age vs gender, glasses vs age).
• StyleGAN’s W-space exhibits stronger disentanglement than Z-space, aiding long-distance attribute edits, while Z-space benefits from conditional projection for decorrelation.
• Real-image editing is feasible: real faces can be edited by inverting to latent codes and applying InterFaceGAN directions, with StyleGAN inversion yielding particularly strong results.
• Artifacts in generated images can be partially corrected by moving latent codes toward a “quality” direction.