[Paper Review] The Unusual Effectiveness of Averaging in GAN Training

Yasin Yazıcı, Chuan-Sheng Foo+4 more|arXiv (Cornell University)|Jun 12, 2018

Advanced Neural Network ApplicationsComputer Science40 references66 citations

TL;DR

The paper investigates moving average (MA) and exponential moving average (EMA) of GAN parameters, showing EMA stabilizes training and improves performance across diverse datasets and architectures, with theoretical insights and extensive experiments.

ABSTRACT

We examine two different techniques for parameter averaging in GAN training. Moving Average (MA) computes the time-average of parameters, whereas Exponential Moving Average (EMA) computes an exponentially discounted sum. Whilst MA is known to lead to convergence in bilinear settings, we provide the -- to our knowledge -- first theoretical arguments in support of EMA. We show that EMA converges to limit cycles around the equilibrium with vanishing amplitude as the discount parameter approaches one for simple bilinear games and also enhances the stability of general GAN training. We establish experimentally that both techniques are strikingly effective in the non-convex-concave GAN setting as well. Both improve inception and FID scores on different architectures and for different GAN objectives. We provide comprehensive experimental results across a range of datasets -- mixture of Gaussians, CIFAR-10, STL-10, CelebA and ImageNet -- to demonstrate its effectiveness. We achieve state-of-the-art results on CIFAR-10 and produce clean CelebA face images.\footnote{~The code is available at \url{https://github.com/yasinyazici/EMA_GAN}}

Motivation & Objective

• Motivate simple, external-averaging strategies to tackle cycling and instability in GAN training without altering the game dynamics.
• Analyze and compare Moving Average (MA) and Exponential Moving Average (EMA) as outside-training-loop regularizers.
• Provide theoretical justification for EMA’s stabilizing effects and demonstrate empirical gains across multiple datasets and GAN objectives.

Proposed method

• Define Moving Average (MA) and Exponential Moving Average (EMA) over generator/discriminator parameters during GAN training.
• Derive EMA update rule theta_EMA^(t) = beta * theta_EMA^(t-1) + (1 - beta) * theta^(t) and contrast with MA.
• Provide a theoretical analysis in bilinear saddle-point settings showing EMA reduces oscillation amplitude without forcing convergence to equilibrium.
• Extend analysis to local stability around Nash equilibria in non-bilinear, non-convex-concave GANs.
• Conduct extensive experiments on mixtures of Gaussians, CIFAR-10, STL-10, CelebA, and ImageNet using original GAN and Wasserstein-1 objectives, with ADAM optimizer and alternating updates.
• Compare EMA/MA against baseline, Optimistic Adam (OMD), Consensus Optimization (CO), and Zero-Centered Gradient Penalty (Zero-GP).

Experimental results

Research questions

• RQ1 Does EMA stabilize cycling around equilibria in GAN training and how does its effect compare to MA in bilinear and non-bilinear settings?
• RQ2 Do EMA and MA improve GAN performance (IS and FID) across different datasets, architectures, and objectives without altering the underlying game dynamics?
• RQ3 How do EMA and MA interact with other stabilization methods (OMD, CO, Zero-GP) on non-convex/concave GANs?
• RQ4 What are the theoretical implications of EMA on the Jacobian dynamics near equilibria in GAN training?
• RQ5 What practical guidance emerges for hyperparameters (beta, start point) and evaluation across datasets?

Key findings

• EMA stabilizes cyclic behavior by shrinking oscillation amplitude around the equilibrium in bilinear games, and preserves stability of locally stable fixed points in non-bilinear settings.
• Experimentally, EMA improves Inception Score and FID across CIFAR-10, STL-10, CelebA, and ImageNet compared to non-averaged models and MA.
• EMA outperforms MA, often with more robust gains across datasets and architectures; MA can degrade performance when averaging over overly long windows.
• On mixture-of-Gaussians, EMA yields lower Wasserstein-1 distance than baselines and other averaging methods, indicating more balanced mode coverage.
• Qualitatively, EMA reduces artifacts and yields smoother evolution of generated images in CelebA and CIFAR-10; MA can introduce degradation at longer averaging windows.
• The study reports state-of-the-art-like results on CIFAR-10 and clean CelebA faces under the tested settings.