[Paper Review] Do Not Escape From the Manifold: Discovering the Local Coordinates on the Latent Space of GANs

Jaewoong Choi, Jun-Ho Lee+4 more|arXiv (Cornell University)|Jun 13, 2021

Generative Adversarial Networks and Image Synthesis27 references9 citations

TL;DR

This paper proposes Local Basis, an unsupervised method that discovers semantic-factorizing traversal directions in the latent space of GANs by analyzing local geometry via tangent spaces of the $ω$-manifold. It shows that $ω$-space in StyleGAN2 is globally warped, explaining the limited success of global methods, and demonstrates superior stability and disentanglement through local traversal and Grassmannian metric analysis.

ABSTRACT

The discovery of the disentanglement properties of the latent space in GANs motivated a lot of research to find the semantically meaningful directions on it. In this paper, we suggest that the disentanglement property is closely related to the geometry of the latent space. In this regard, we propose an unsupervised method for finding the semantic-factorizing directions on the intermediate latent space of GANs based on the local geometry. Intuitively, our proposed method, called Local Basis, finds the principal variation of the latent space in the neighborhood of the base latent variable. Experimental results show that the local principal variation corresponds to the semantic factorization and traversing along it provides strong robustness to image traversal. Moreover, we suggest an explanation for the limited success in finding the global traversal directions in the latent space, especially W-space of StyleGAN2. We show that W-space is warped globally by comparing the local geometry, discovered from Local Basis, through the metric on Grassmannian Manifold. The global warpage implies that the latent space is not well-aligned globally and therefore the global traversal directions are bound to show limited success on it.

Motivation & Objective

• To address the challenge of finding robust, semantically meaningful traversal directions in GAN latent spaces, especially where global methods fail.
• To investigate whether the disentanglement properties of GAN latent spaces are fundamentally tied to their geometric structure.
• To develop an unsupervised method that identifies local, stable traversal directions based on the intrinsic geometry of the latent manifold.
• To evaluate the global geometry of the $ω$-space in StyleGAN2 using local information and propose a metric for global disentanglement.
• To explain the limited success of global methods like GANSpace and SeFa by quantifying global warping in the latent space.

Proposed method

• Local Basis is defined as an ordered orthonormal basis of the tangent space $T_{\mathbf{w}}\mathcal{W}$ at a latent code $\mathbf{w}$, representing the local principal variation of the latent manifold.
• The method computes the singular vectors of the Jacobian matrix of the generator network, which provide a closed-form solution for the local basis.
• It uses Grassmannian manifold metrics to compare local bases across different latent codes, enabling quantification of geometric consistency and warping.
• The iterative curve-traversal method traces curved paths along the latent manifold using Local Basis, improving image traversal stability over linear interpolation.
• Global disentanglement is evaluated by comparing local bases from nearby and distant latent codes using projection and geodesic metrics on the Grassmannian.

Experimental results

Research questions

• RQ1 Is the disentanglement of semantics in GAN latent spaces fundamentally linked to the underlying geometric structure of the manifold?
• RQ2 Why do global traversal methods like GANSpace and SeFa show limited success despite promising results in localized regions?
• RQ3 Can local geometric analysis reveal a more stable and semantically meaningful traversal direction than global methods?
• RQ4 To what extent is the $ω$-space of StyleGAN2 globally warped, and how does this affect the performance of global basis methods?
• RQ5 Can iterative curve-traversal using local bases produce more stable image transformations than linear interpolation?

Key findings

• The Grassmannian metric shows that Local Basis from nearby latent codes ($|\mathbf{z}' - \mathbf{z}| = 0.1$) are significantly more consistent than those from random codes, indicating local geometric stability.
• The metric between random $\mathbf{w}$ and random $O(d_{\mathcal{W}})$ is much larger than between close $\mathbf{w}$, proving that $\mathcal{W}$-space is globally warped.
• Local Basis achieves more stable and semantically meaningful image traversal than global methods like GANSpace and SeFa, especially in terms of image quality and semantic consistency.
• The iterative curve-traversal method based on Local Basis produces smoother and more coherent image sequences than linear traversal, demonstrating improved robustness.
• The global basis from GANSpace and SeFa shows only limited alignment with the local geometry, as evidenced by high metric scores relative to close $\mathbf{w}$, indicating incomplete global disentanglement.
• The analysis confirms that the limited success of global methods is due to the global warping of $\mathcal{W}$-space, which disrupts consistent semantic directionality across the latent space.