[Paper Review] On the Effects of Batch and Weight Normalization in Generative Adversarial Networks

Sitao Xiang, Hao Li|arXiv (Cornell University)|Apr 13, 2017

Generative Adversarial Networks and Image Synthesis14 references70 citations

TL;DR

The paper evaluates Batch Normalization versus a proposed Weight Normalization for GANs, introduces a translated ReLU-based weight-normalized approach, and shows WN improves stability and reconstruction quality over BN in DCGAN-like architectures.

ABSTRACT

Generative adversarial networks (GANs) are highly effective unsupervised learning frameworks that can generate very sharp data, even for data such as images with complex, highly multimodal distributions. However GANs are known to be very hard to train, suffering from problems such as mode collapse and disturbing visual artifacts. Batch normalization (BN) techniques have been introduced to address the training. Though BN accelerates the training in the beginning, our experiments show that the use of BN can be unstable and negatively impact the quality of the trained model. The evaluation of BN and numerous other recent schemes for improving GAN training is hindered by the lack of an effective objective quality measure for GAN models. To address these issues, we first introduce a weight normalization (WN) approach for GAN training that significantly improves the stability, efficiency and the quality of the generated samples. To allow a methodical evaluation, we introduce squared Euclidean reconstruction error on a test set as a new objective measure, to assess training performance in terms of speed, stability, and quality of generated samples. Our experiments with a standard DCGAN architecture on commonly used datasets (CelebA, LSUN bedroom, and CIFAR-10) indicate that training using WN is generally superior to BN for GANs, achieving 10% lower mean squared loss for reconstruction and significantly better qualitative results than BN. We further demonstrate the stability of WN on a 21-layer ResNet trained with the CelebA data set. The code for this paper is available at https://github.com/stormraiser/gan-weightnorm-resnet

Motivation & Objective

• Assess how Batch Normalization impacts GAN training stability and sample quality.
• Propose a weight normalization-based training variant for GANs and assess its benefits.
• Introduce an objective reconstruction-based metric to evaluate GAN training speed, stability, and quality.
• Demonstrate the approach on DCGAN architectures across CelebA, LSUN, and CIFAR-10 datasets.

Proposed method

• Introduce a modified Weight Normalization (WN) technique for GAN training and compare it with Batch Normalization (BN).
• Define a translated ReLU (TReLU) within weight-normalized layers to preserve representational capacity without introducing large mean shifts.
• Propose an objective reconstruction loss on a test set, using gradient descent to find latent codes that minimize ||G(z)-x||^2.
• Use RMSProp for optimization and conduct experiments with DCGAN-like architectures on CelebA, LSUN bedroom, and CIFAR-10, plus a 21-layer ResNet experiment.
• Evaluate reconstruction quality via final reconstruction loss and training stability via evolution of generated samples over iterations.

Experimental results

Research questions

• RQ1 Does Batch Normalization improve or degrade GAN training stability and sample quality across standard benchmarks?
• RQ2 Is a Weight Normalization formulation superior to BN for GAN training in terms of stability and reconstruction quality?
• RQ3 Can a reconstruction-based objective reliably monitor GAN training progress and reflect qualitative sample quality?
• RQ4 How does the proposed WN approach perform on deeper architectures (e.g., ResNet) and across multiple datasets?

Key findings

• Weight Normalization achieves about 10.5% lower final reconstruction loss than the vanilla model.
• Weight Normalization yields more stable training and better reconstruction quality than Batch Normalization.
• Batch Normalization can cause instability and artifacts, while vanilla GANs may collapse on some datasets.
• A 21-layer ResNet GAN trained with Weight Normalization remains stable and produces diverse, high-quality samples.
• BN accelerates early training but can reduce long-term stability and sample diversity compared to WN.
• WN provides faster and more stable training than BN, with equal or higher sample quality than non-normalized models.