[Paper Review] Collaborative Learning for Faster StyleGAN Embedding

Shanyan Guan, Ying Tai+4 more|arXiv (Cornell University)|Jul 3, 2020

Generative Adversarial Networks and Image SynthesisComputer Science40 references68 citations

TL;DR

The paper introduces a collaborative learning framework that jointly trains an embedding network and an optimization-based iterator to efficiently embed real images into StyleGAN's latent space, achieving real-time inference with competitive inversion quality.

ABSTRACT

The latent code of the recent popular model StyleGAN has learned disentangled representations thanks to the multi-layer style-based generator. Embedding a given image back to the latent space of StyleGAN enables wide interesting semantic image editing applications. Although previous works are able to yield impressive inversion results based on an optimization framework, which however suffers from the efficiency issue. In this work, we propose a novel collaborative learning framework that consists of an efficient embedding network and an optimization-based iterator. On one hand, with the progress of training, the embedding network gives a reasonable latent code initialization for the iterator. On the other hand, the updated latent code from the iterator in turn supervises the embedding network. In the end, high-quality latent code can be obtained efficiently with a single forward pass through our embedding network. Extensive experiments demonstrate the effectiveness and efficiency of our work.

Motivation & Objective

• Motivate efficient inversion of real images into StyleGAN latent space for real-time editing.
• Develop an embedding network that disentangles identity and attributes to map images to W+ latent codes.
• Leverage a collaborative loop where iterator refinements supervise the embedding network.
• Achieve fast, high-quality inversion without requiring paired latent codes or offline optimization.
• Demonstrate broad semantic editing applications enabled by fast embedding.

Proposed method

• Propose a collaborative framework with an embedding network and an optimization-based iterator for latent code inversion.
• Use two encoders (identity and attribute) whose features are merged via denormalization to predict w_e in W+.
• Initialize the iterator with w_e and optimize to w_o using a loss L_opt combining MSE and LPIPS.
• Supervise the embedding network with losses on latent codes (L_w), image (L_mse), and perceptual (L_per) signals.
• Iterate online with a cache mechanism to preserve the best supervision and accelerate convergence.

Experimental results

Research questions

• RQ1 Can an embedding network paired with an optimization-based iterator produce high-quality StyleGAN inversions faster than offline optimization?
• RQ2 Does disentangling identity and attribute information in the embedding network improve latent code accuracy and editing quality?
• RQ3 How does collaborative learning influence convergence speed and inversion metrics (PSNR, SSIM, LPIPS) compared to state-of-the-art methods?

Key findings

• The method achieves competitive inversion quality while being roughly 500x faster than the fastest prior approach.
• Ours achieves LPIPS of 0.16 (CelebA-HQ) and 0.11 (CACD), with PSNR 31.47 (CelebA-HQ) and 32.05 (CACD), and SSIM 0.83 (both datasets).
• The iterator benefits from a better initialization provided by the embedding network, leading to faster convergence and better upper-bound performance.
• Disentangled identity and attribute encoders improve inversion quality over single ResNet-based encoders.
• A cache mechanism ensures the embedding network receives strong supervisory signals even when the iterator’s recent results are suboptimal.