[Paper Review] Demystifying MMD GANs

Mikołaj Bińkowski, Danica J. Sutherland+2 more|arXiv (Cornell University)|Jan 4, 2018

Model Reduction and Neural NetworksPhysics and Astronomy61 references99 citations

TL;DR

The paper analyzes MMD GANs, showing unbiased generator gradients with fixed representations but biased gradients when the critic is learned, and demonstrates practical advantages over WGAN-GP.

ABSTRACT

We investigate the training and performance of generative adversarial networks using the Maximum Mean Discrepancy (MMD) as critic, termed MMD GANs. As our main theoretical contribution, we clarify the situation with bias in GAN loss functions raised by recent work: we show that gradient estimators used in the optimization process for both MMD GANs and Wasserstein GANs are unbiased, but learning a discriminator based on samples leads to biased gradients for the generator parameters. We also discuss the issue of kernel choice for the MMD critic, and characterize the kernel corresponding to the energy distance used for the Cramer GAN critic. Being an integral probability metric, the MMD benefits from training strategies recently developed for Wasserstein GANs. In experiments, the MMD GAN is able to employ a smaller critic network than the Wasserstein GAN, resulting in a simpler and faster-training algorithm with matching performance. We also propose an improved measure of GAN convergence, the Kernel Inception Distance, and show how to use it to dynamically adapt learning rates during GAN training.

Motivation & Objective

• Clarify gradient bias in MMD GANs and compare to Wasserstein GANs.
• Investigate kernel choices and their impact on the MMD critic.
• Relate energy distance and Cramér GAN to MMD, including gradient considerations.
• Propose practical evaluation metrics like Kernel Inception Distance (KID).
• Demonstrate training benefits of MMD GANs with smaller critics and faster training.

Proposed method

• Formulate MMD as an IPM with the unit ball in an RKHS as witness class.
• Regularize the MMD critic via gradient penalties similar to WGAN-GP.
• Connect energy distance and Cramér GAN to MMD via kernel constructions.
• Develop Kernel Inception Distance (KID) as an unbiased GAN convergence metric.
• Empirically compare MMD GANs to WGAN-GP and Cramér GAN on standard datasets.

Experimental results

Research questions

• RQ1 Do gradient estimators for MMD GANs produce unbiased generator gradients when the critic is fixed versus learned?
• RQ2 How does kernel choice affect MMD critic performance and training stability?
• RQ3 Can energy distance/Cramér GAN insights improve MMD GANs and related IPMs?
• RQ4 Is Kernel Inception Distance a reliable, unbiased convergence measure for GANs?
• RQ5 Do MMD GANs achieve comparable performance with smaller critics and faster training than WGAN-GP?

Key findings

• The natural MMD estimator has unbiased gradients when used on top of a fixed deep representation.
• Learning the critic makes generator gradients biased with respect to the infinite-sample optimum.
• MMD GANs can match WGAN-GP performance with smaller critic networks and faster training.
• The energy distance connections provide a gradient-penalized, regularized critic framework.
• Kernel Inception Distance (KID) is proposed as an unbiased convergence metric and aids learning-rate adaptation.
• Experiments on standard benchmarks show MMD GANs’ practical advantages over WGAN-GP in network size and training efficiency.