[Paper Review] Cumulant GAN.

Yannis Pantazis, Dipjyoti Paul+3 more|arXiv (Cornell University)|Jun 11, 2020

Generative Adversarial Networks and Image Synthesis62 references4 citations

TL;DR

This paper introduces Cumulant GAN, a novel GAN loss function based on cumulant generating functions that unifies Rényi divergence minimization, encompassing KLD, reverse KLD, Hellinger distance, and χ²-divergence. It proves linear convergence to Nash equilibrium under linear discriminators with Gaussian data and demonstrates improved image generation stability and performance over Wasserstein GAN in inception score and Fréchet inception distance.

ABSTRACT

In this paper, we propose a novel loss function for training Generative Adversarial Networks (GANs) aiming towards deeper theoretical understanding as well as improved stability and performance for the underlying optimization problem. The new loss function is based on cumulant generating functions giving rise to \emph{Cumulant GAN}. Relying on a recently-derived variational formula, we show that the corresponding optimization problem is equivalent to R{\'e}nyi divergence minimization, thus offering a (partially) unified perspective of GAN losses: the R{\'e}nyi family encompasses Kullback-Leibler divergence (KLD), reverse KLD, Hellinger distance and $\chi^2$-divergence. Wasserstein GAN is also a member of cumulant GAN. In terms of stability, we rigorously prove the linear convergence of cumulant GAN to the Nash equilibrium for a linear discriminator, Gaussian distributions and the standard gradient descent ascent algorithm. Finally, we experimentally demonstrate that image generation is more robust relative to Wasserstein GAN and it is substantially improved in terms of both inception score and Fr\'echet inception distance when both weaker and stronger discriminators are considered.

Motivation & Objective

• To develop a theoretically grounded GAN loss function that enhances training stability and performance.
• To unify existing GAN losses under the Rényi divergence framework via a variational formula.
• To establish theoretical convergence guarantees for the optimization process in GAN training.
• To empirically validate improved generative quality across diverse discriminator strengths.

Proposed method

• The method employs a variational formula derived from cumulant generating functions to define a new GAN loss function.
• The loss function is shown to be equivalent to minimizing Rényi divergence, unifying multiple existing GAN objectives.
• Theoretical analysis proves linear convergence to Nash equilibrium under linear discriminators with Gaussian data using standard gradient descent ascent.
• The framework includes Wasserstein GAN as a special case, extending its theoretical foundation.
• The approach enables stable training across both weak and strong discriminators.
• Empirical evaluation uses inception score and Fréchet inception distance to assess image generation quality.

Experimental results

Research questions

• RQ1 Can a unified theoretical framework be established for GAN losses using cumulant generating functions?
• RQ2 Does the proposed cumulant-based loss ensure linear convergence to Nash equilibrium under standard training dynamics?
• RQ3 How does the performance of Cumulant GAN compare to Wasserstein GAN in terms of image quality metrics?
• RQ4 Can the method maintain stability and improve performance when using weaker or stronger discriminators?
• RQ5 Is the Rényi divergence family fully encompassed within the proposed cumulant GAN framework?

Key findings

• Cumulant GAN unifies multiple GAN losses, including Kullback-Leibler divergence, reverse KLD, Hellinger distance, and χ²-divergence, under the Rényi divergence minimization framework.
• The method proves linear convergence to Nash equilibrium for linear discriminators with Gaussian data under standard gradient descent ascent.
• Image generation with Cumulant GAN shows improved robustness compared to Wasserstein GAN across both weak and strong discriminator settings.
• Cumulant GAN achieves substantially better inception scores and Fréchet inception distances than Wasserstein GAN.
• Wasserstein GAN is formally shown to be a special case within the Cumulant GAN framework, extending its theoretical basis.
• The proposed loss function enhances training stability and generative performance without requiring architectural modifications.