[Paper Review] Spectral Normalization for Generative Adversarial Networks

Takeru Miyato, Toshiki Kataoka+2 more|arXiv (Cornell University)|Feb 16, 2018

Generative Adversarial Networks and Image Synthesis24 references2,304 citations

TL;DR

Introduces spectral normalization to stabilize GAN training by constraining each layer’s spectral norm, yielding robust performance across datasets like CIFAR-10, STL-10, and ImageNet. SN-GANs achieve competitive or superior image quality with simple integration and low overhead.

ABSTRACT

One of the challenges in the study of generative adversarial networks is the instability of its training. In this paper, we propose a novel weight normalization technique called spectral normalization to stabilize the training of the discriminator. Our new normalization technique is computationally light and easy to incorporate into existing implementations. We tested the efficacy of spectral normalization on CIFAR10, STL-10, and ILSVRC2012 dataset, and we experimentally confirmed that spectrally normalized GANs (SN-GANs) is capable of generating images of better or equal quality relative to the previous training stabilization techniques.

Motivation & Objective

• Motivate and address instability in GAN training by constraining the discriminator.
• Propose a lightweight normalization that sets layer spectral norms to 1 to bound the discriminator’s Lipschitz constant.
• Demonstrate efficacy of spectral normalization (SN) across multiple datasets and compare with existing regularization techniques.

Proposed method

• Define a discriminator as a composition of linear layers and nonlinearities; constrain the Lipschitz constant by normalizing each weight matrix W with its spectral norm: W_bar_SN = W / sigma(W).
• Estimate sigma(W) efficiently with the power iteration method to keep computation lightweight.
• Analyze gradient flow under spectral normalization to show an adaptive regularization effect on the dominant singular components.
• Compare SN with weight normalization, weight clipping, gradient penalty, and orthonormal regularization both theoretically and empirically.
• Evaluate SN-GANs on CIFAR-10, STL-10, and ImageNet using standard GAN objectives and hinge loss variants; report Inception Score and FID as evaluation metrics.

Experimental results

Research questions

• RQ1 Does spectral normalization reliably bound the discriminator’s Lipschitz constant without excessive computational cost?
• RQ2 How does SN compare to other regularization techniques (weight normalization, weight clipping, gradient penalty, orthonormal regularization) in terms of image quality and training stability?
• RQ3 Is SN robust across architectures and datasets (CIFAR-10, STL-10, ImageNet) and with different optimization settings?
• RQ4 What is the effect of spectral normalization on the discriminator’s weight spectra and feature utilization?

Key findings

• SN constrains the discriminator’s Lipschitz constant by normalizing each layer’s spectral norm, maintaining bounded gradients and improving stability.
• Empirical results show SN-GANs achieve higher or comparable image quality to prior stabilization methods across CIFAR-10, STL-10, and ImageNet, often with faster or comparable training times.
• Spectral normalization yields more broadly distributed layer singular values compared to weight clipping/normalization, indicating richer feature utilization in early layers.
• SN-GANs exhibit robustness to aggressive learning rates and momentum settings, outperforming WGAN-GP under several settings.
• Compared to orthonormal regularization, SN-GANs maintain performance across varying feature-map dimensions and architectures, indicating greater architectural robustness.