[Paper Review] LOGAN: Latent Optimisation for Generative Adversarial Networks

Yan Wu, Jeff Donahue+3 more|arXiv (Cornell University)|Dec 2, 2019

Generative Adversarial Networks and Image Synthesis48 references54 citations

TL;DR

LOGAN improves GAN training by performing natural-gradient-based latent optimisation of the input z, enhancing discriminator-generator interaction and achieving state-of-the-art ImageNet (128x128) results without architectural changes.

ABSTRACT

Training generative adversarial networks requires balancing of delicate adversarial dynamics. Even with careful tuning, training may diverge or end up in a bad equilibrium with dropped modes. In this work, we improve CS-GAN with natural gradient-based latent optimisation and show that it improves adversarial dynamics by enhancing interactions between the discriminator and the generator. Our experiments demonstrate that latent optimisation can significantly improve GAN training, obtaining state-of-the-art performance for the ImageNet ($128 imes 128$) dataset. Our model achieves an Inception Score (IS) of $148$ and an Fréchet Inception Distance (FID) of $3.4$, an improvement of $17\%$ and $32\%$ in IS and FID respectively, compared with the baseline BigGAN-deep model with the same architecture and number of parameters.

Motivation & Objective

• Improve the adversarial dynamics in GAN training by optimising the latent input z during training.
• Demonstrate that latent optimisation via natural gradient descent (NGD) yields higher image fidelity and diversity on large-scale GANs.
• Provide theoretical insight into how latent optimisation affects differentiable game dynamics in GANs.
• Show that LOGAN can surpass state-of-the-art BigGAN-deep without changing architecture.
• Analyze the relationship of LOGAN to Unrolled GANs and Symplectic Gradient Adjustment (SGA).

Proposed method

• Use a latent optimisation step to update z via gradients of the generator loss with respect to z.
• Replace standard gradient descent with natural gradient descent to compute the latent update Δz (NGD), leading to Δz = α g/(β + ||g||^2) where g = ∂f(z)/∂z.
• Backpropagate through the latent optimisation to obtain second-order terms that couple D and G dynamics.
• Regularise latent updates with a z-regularisation term Rz and optionally optimise a portion c of z while keeping some elements random.
• Evaluate LOGAN on medium-scale DCGAN/SN-GAN and large-scale BigGAN-deep on ImageNet (128x128) with and without truncation curves.
• Compare against baseline BigGAN-deep and LOGAN variants (GD and NGD) using FID and IS metrics.

Experimental results

Research questions

• RQ1 Can latent optimisation of the latent code z via natural gradient descent improve GAN training dynamics and sample quality on large-scale models?
• RQ2 How does LOGAN compare to baseline BigGAN-deep and to latent optimisation with gradient descent (GD) in terms of FID and IS?
• RQ3 What theoretical connections exist between LOGAN, SGA, and Unrolled GANs, and how do these inform the dynamics of adversarial training?
• RQ4 What are the practical considerations (hyperparameters, regularisation, evaluation) for scaling LOGAN to ImageNet-scale generation?

Key findings

• LOGAN with NGD achieves substantial improvements over baseline BigGAN-deep on ImageNet 128x128, with FID 3.36±0.14 and IS 148.2±3.1.
• Compared to BigGAN-deep, LOGAN-NGD reduces FID by about 32% and increases IS by about 17%.
• LOGAN-GD shows smaller gains than LOGAN-NGD, underscoring the benefit of NGD over simple gradient updates for z.
• Latent optimisation via LOGAN improves training dynamics by introducing second-order interactions akin to SGA, while avoiding full unrolling of network parameters.
• Regularising and partially updating z (e.g., 50-80% of z) with appropriate damping and a z-regulariser stabilises training and enhances sample quality.
• In ablation studies, removing latent-derivative terms or using stop_gradient degrades stability, confirming the importance of backpropagated second-order terms through latent optimisation.