[Paper Review] Disentangled Inference for GANs with Latently Invertible Autoencoder

Jiapeng Zhu, Deli Zhao+2 more|arXiv (Cornell University)|Jun 19, 2019

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes Latently Invertible Autoencoder (LIA), a two-stage method that enables disentangled inference in GANs by embedding a symmetric invertible network in the latent space. By training the GAN decoder first and then learning a disentangled encoder via a separate autoencoder, LIA achieves superior image reconstruction and faster convergence compared to prior methods, with state-of-the-art results on FFHQ and LSUN datasets.

ABSTRACT

Generative Adversarial Networks (GANs) play an increasingly important role in machine learning. However, there is one fundamental issue hindering their practical applications: the absence of capability for encoding real-world samples. The conventional way of addressing this issue is to learn an encoder for GAN via Variational Auto-Encoder (VAE). In this paper, we show that the entanglement of the latent space for the VAE/GAN framework poses the main challenge for encoder learning. To address the entanglement issue and enable inference in GAN we propose a novel algorithm named Latently Invertible Autoencoder (LIA). The framework of LIA is that an invertible network and its inverse mapping are symmetrically embedded in the latent space of VAE. The decoder of LIA is first trained as a standard GAN with the invertible network and then the partial encoder is learned from a disentangled autoencoder by detaching the invertible network from LIA, thus avoiding the entanglement problem caused by the random latent space. Experiments conducted on the FFHQ face dataset and three LSUN datasets validate the effectiveness of LIA/GAN.

Motivation & Objective

• To address the fundamental challenge of GAN inference, where real images cannot be reliably encoded into the latent space.
• To identify the entanglement of the latent space (z-space) as the key reason for poor performance in VAE-based GAN inference methods.
• To develop a method that enables accurate, disentangled latent code inference for GANs without compromising image generation quality.
• To design a two-stage training scheme that decouples decoder pretraining from encoder learning, avoiding entanglement issues.

Proposed method

• LIA embeds an invertible network and its inverse symmetrically within the latent space of an autoencoder, enabling bidirectional mapping between latent codes and image features.
• The decoder is first pre-trained as a standard GAN using the invertible network to map from latent space to image space.
• The encoder is then learned from a disentangled autoencoder by detaching the invertible network, ensuring the learned latent codes are disentangled.
• The method uses a two-stage training process: first train the GAN with the invertible network, then train the encoder on the disentangled latent space.
• The invertible network allows for exact reconstruction and enables initialization of latent codes via the encoder, improving optimization stability.
• The framework supports replacing the decoder with any GAN generator, enabling diverse applications like image editing and data augmentation.

Experimental results

Research questions

• RQ1 Why do VAE-based GAN inference methods fail to achieve high reconstruction quality despite using variational inference?
• RQ2 How does the disentanglement of the intermediate latent space (y-space) compare to the entanglement of the original latent space (z-space) in terms of inference performance?
• RQ3 Can a symmetric invertible network in the latent space improve the accuracy and stability of GAN inversion?
• RQ4 Does using the encoder output from LIA as initialization for optimization lead to faster convergence and better reconstruction than random or mean-based initialization?
• RQ5 Can the two-stage training scheme effectively decouple GAN generation from encoder learning while preserving disentanglement?

Key findings

• LIA achieves the best reconstruction quality on the FFHQ dataset, with the highest FID and LPIPS scores among compared methods.
• Using the encoder output from LIA as initialization leads to faster convergence and lower reconstruction loss during optimization, as shown in Figure 12.
• The method significantly outperforms both random initialization and mean-based initialization (used in Image2StyleGAN) in terms of reconstruction fidelity and convergence speed.
• Qualitative results in Figure 9 and Table 4 confirm that LIA recovers fine details such as Mona Lisa’s hairstyle more accurately than other methods.
• Experiments in the y-space (disentangled) show superior convergence and reconstruction compared to optimization in the z-space (entangled), proving that disentanglement is crucial for successful GAN inference.
• The two-stage training scheme successfully avoids entanglement issues, enabling high-quality encoder learning without compromising the GAN’s generation capability.