[Paper Review] Stabilizing Generative Adversarial Networks: A Survey

Maciej Wiatrak, Stefano V. Albrecht+1 more|arXiv (Cornell University)|Sep 30, 2019

Model Reduction and Neural NetworksPhysics and Astronomy87 references65 citations

TL;DR

A comprehensive survey of GAN training stabilization methods, categorizing approaches into architecture, loss, game theory, multi-agent, and gradient-based methods, and outlining open problems.

ABSTRACT

Generative Adversarial Networks (GANs) are a type of generative model which have received much attention due to their ability to model complex real-world data. Despite their recent successes, the process of training GANs remains challenging, suffering from instability problems such as non-convergence, vanishing or exploding gradients, and mode collapse. In recent years, a diverse set of approaches have been proposed which focus on stabilizing the GAN training procedure. The purpose of this survey is to provide a comprehensive overview of the GAN training stabilization methods which can be found in the literature. We discuss the advantages and disadvantages of each approach, offer a comparative summary, and conclude with a discussion of open problems.

Motivation & Objective

• Provide a holistic taxonomy of GAN stabilization methods.
• Assess the advantages and limitations of each stabilization approach.
• Summarize comparative insights and practical considerations for stabilizing GAN training.
• Highlight open problems and directions for future research in GAN stabilization.

Proposed method

• Categorize stabilization methods into five families: architecture, loss functions, game theory, multi-agent, and gradient-based approaches.
• Describe representative techniques within each category (e.g., DCGAN, SAGAN, PROGAN; various f-divergence and IPM losses; MNE and mixed-strategy equilibria; multi-generator/discriminator setups).
• Discuss practical trade-offs, computational costs, and complementarities between methods.
• Provide a comparative summary and discuss open problems in GAN stabilization.

Experimental results

Research questions

• RQ1 What are the main sources of instability in GAN training?
• RQ2 What stabilization strategies exist and how do they address convergence, vanishing/exploding gradients, and mode collapse?
• RQ3 What are the trade-offs and limitations of architecture-, loss-, game theory-, multi-agent-, and gradient-based stabilization methods?
• RQ4 What open problems remain for achieving robust, off-the-shelf GAN training?
• RQ5 How do different stabilization approaches complement each other in practice?

Key findings

• Architecture variants (e.g., DCGAN, SAGAN, BigGAN) can improve stability and sample quality but may increase computational cost.
• Loss-function based approaches (f-divergences, IPMs like Wasserstein, MMD) offer improved training dynamics but often require specific constraints and tuning.
• Game-theoretic and multi-agent formulations provide theoretical convergence insights and can help with mode collapse, though practical applicability is limited by assumptions and costs.
• Gradient-based optimization refinements (e.g., Optimistic Mirror Descent, ConOpt, Competitive Gradient Descent) address non-convex dynamics but have restricted applicability or require additional mechanisms.
• BigGAN represents a strong baseline across aspects but at a high computational cost, illustrating the trade-off between stability gains and resources.