[Paper Review] The GAN Landscape: Losses, Architectures, Regularization, and Normalization

Karol Kurach, Mario Lučić+3 more|arXiv (Cornell University)|Jun 5, 2018

Generative Adversarial Networks and Image SynthesisComputer Science37 references124 citations

TL;DR

This paper provides a comprehensive empirical study of the GAN landscape, systematically evaluating losses, architectures, regularization, and normalization techniques. It reproduces state-of-the-art results, identifies common training pitfalls, and releases code and pre-trained models to improve reproducibility and guide future GAN development.

ABSTRACT

Generative adversarial networks (GANs) are a class of deep generative models which aim to learn a target distribution in an unsupervised fashion. While they were successfully applied to many problems, training a GAN is a notoriously challenging task and requires a significant amount of hyperparameter tuning, neural architecture engineering, and a non-trivial amount of tricks. The success in many practical applications coupled with the lack of a measure to quantify the failure modes of GANs resulted in a plethora of proposed losses, regularization and normalization schemes, and neural architectures. In this work we take a sober view of the current state of GANs from a practical perspective. We reproduce the current state of the art and go beyond fairly exploring the GAN landscape. We discuss common pitfalls and reproducibility issues, open-source our code on Github, and provide pre-trained models on TensorFlow Hub.

Motivation & Objective

• To provide a systematic, reproducible evaluation of current GAN training practices.
• To identify common failure modes and hyperparameter pitfalls in GAN training.
• To establish a benchmark for state-of-the-art GAN performance using standardized protocols.
• To promote reproducibility by open-sourcing code and pre-trained models on TensorFlow Hub.

Proposed method

• Empirically evaluate a wide range of GAN losses, including standard GAN, WGAN, hinge, and LS-GAN, under consistent training protocols.
• Systematically test various neural network architectures, normalization layers (BatchNorm, InstanceNorm, LayerNorm), and regularization techniques.
• Apply and compare different normalization schemes across different architectures and training setups.
• Use controlled ablation studies to isolate the impact of each component on training stability and sample quality.
• Reproduce state-of-the-art results on CIFAR-10 and CelebA datasets using standardized training procedures.
• Release code and pre-trained models on GitHub and TensorFlow Hub to ensure transparency and reproducibility.

Experimental results

Research questions

• RQ1 Which GAN loss function yields the most stable training and best sample quality across different datasets and architectures?
• RQ2 How do different normalization layers (BatchNorm, InstanceNorm, LayerNorm) affect GAN training stability and performance?
• RQ3 What is the impact of architectural choices and regularization techniques on GAN convergence and mode coverage?
• RQ4 To what extent can state-of-the-art GAN performance be consistently reproduced using standardized training protocols?
• RQ5 What are the most common reproducibility pitfalls in GAN training, and how can they be mitigated?

Key findings

• The choice of GAN loss significantly affects training stability and sample quality, with hinge and WGAN losses showing superior performance on CIFAR-10 and CelebA.
• Normalization layers such as InstanceNorm and LayerNorm often outperform BatchNorm in GAN training, particularly on smaller datasets.
• Architectural choices, especially residual connections and self-attention modules, play a crucial role in improving sample diversity and fidelity.
• Reproducibility is highly sensitive to hyperparameters and training protocols, with small changes leading to large performance variations.
• The use of standardized training procedures and open-sourced code enables consistent reproduction of state-of-the-art results.
• Pre-trained models released on TensorFlow Hub facilitate benchmarking and downstream model adaptation.