[Paper Review] Training generative neural networks via Maximum Mean Discrepancy optimization

Gintare Karolina Dziugaite, Daniel M. Roy+1 more|arXiv (Cornell University)|May 14, 2015

Generative Adversarial Networks and Image SynthesisComputer Science8 references222 citations

TL;DR

This paper proposes MMD nets, a method for training deep generative neural networks by minimizing the Maximum Mean Discrepancy (MMD) between generated and real data distributions using a nonparametric two-sample test. Unlike GANs that rely on adversarial training, MMD nets use an unbiased empirical estimate of MMD as a differentiable objective, enabling stable optimization with theoretical generalization bounds.

ABSTRACT

We consider training a deep neural network to generate samples from an unknown distribution given i.i.d. data. We frame learning as an optimization minimizing a two-sample test statistic---informally speaking, a good generator network produces samples that cause a two-sample test to fail to reject the null hypothesis. As our two-sample test statistic, we use an unbiased estimate of the maximum mean discrepancy, which is the centerpiece of the nonparametric kernel two-sample test proposed by Gretton et al. (2012). We compare to the adversarial nets framework introduced by Goodfellow et al. (2014), in which learning is a two-player game between a generator network and an adversarial discriminator network, both trained to outwit the other. From this perspective, the MMD statistic plays the role of the discriminator. In addition to empirical comparisons, we prove bounds on the generalization error incurred by optimizing the empirical MMD.

Motivation & Objective

• To develop a stable alternative to adversarial training for deep generative models by replacing the discriminator with a nonparametric two-sample test.
• To frame generative model learning as minimizing the empirical MMD between generated and real data distributions.
• To provide theoretical bounds on the generalization error incurred by optimizing the empirical MMD instead of the true population MMD.
• To demonstrate that MMD-based optimization avoids the training instability often seen in GANs while maintaining strong sample quality.

Proposed method

• Formulate the generative model training problem as minimizing the MMD between the data distribution and the generator's output distribution.
• Use an unbiased estimator of the Maximum Mean Discrepancy (MMD) as the training objective, derived from the kernel two-sample test.
• Optimize the generator parameters via gradient descent on the empirical MMD, treating MMD as a differentiable loss function.
• Apply McDiarmid’s inequality and Rademacher complexity bounds to derive generalization error bounds for the empirical MMD estimator.
• Use a universal reproducing kernel Hilbert space (RKHS) with a characteristic kernel to ensure MMD is zero only when distributions are equal.
• Establish theoretical convergence guarantees by bounding the estimation error between empirical and true MMD under various moment conditions.

Experimental results

Research questions

• RQ1 Can MMD-based optimization serve as a stable, non-adversarial alternative to GAN training for deep generative models?
• RQ2 How does the generalization error of the empirical MMD estimator scale with sample size and kernel properties?
• RQ3 What theoretical guarantees can be provided for the convergence of MMD-based training in terms of distributional discrepancy?
• RQ4 How does MMD net performance compare to adversarial nets in terms of sample quality and training stability on synthetic and real data?
• RQ5 Under what conditions does the MMD-based objective ensure that the generator learns the true data distribution?

Key findings

• The MMD net framework achieves stable training by replacing the adversarial discriminator with a closed-form MMD statistic, avoiding mode collapse and training instability common in GANs.
• Theoretical bounds show that the generalization error of the empirical MMD estimator decays as $ O(M^{-1/2}) $ for $ p < 2 $, $ O(M^{-1/2} ext{log}^{3/2}(M)) $ for $ p = 2 $, and $ O(M^{-1/p}) $ for $ p > 2 $, where $ M $ is the sample size.
• The approximation error is zero in the nonparametric limit when the generator family is rich enough and the kernel is characteristic, ensuring that MMD = 0 implies distributional equality.
• Empirical results on synthetic and real data show that MMD nets successfully align the generator output distribution with the true data distribution, as evidenced by decreasing MMD values over training iterations.
• The method achieves competitive sample quality compared to GANs, with no need for alternating discriminator and generator updates, simplifying training dynamics.
• Theoretical analysis confirms that the empirical MMD estimator concentrates around the true MMD with high probability, with tail bounds decaying exponentially in sample size.