[Paper Review] MMD GAN: Towards Deeper Understanding of Moment Matching Network

Chunliang Li, Wei-Cheng Chang+3 more|arXiv (Cornell University)|May 24, 2017

Generative Adversarial Networks and Image Synthesis21 references255 citations

TL;DR

MMD GAN enhances Generative Moment Matching Networks by adversarial kernel learning, achieving competitive results with GANs on benchmarks like MNIST, CIFAR-10, CelebA, and LSUN while using smaller batch sizes.

ABSTRACT

Generative moment matching network (GMMN) is a deep generative model that differs from Generative Adversarial Network (GAN) by replacing the discriminator in GAN with a two-sample test based on kernel maximum mean discrepancy (MMD). Although some theoretical guarantees of MMD have been studied, the empirical performance of GMMN is still not as competitive as that of GAN on challenging and large benchmark datasets. The computational efficiency of GMMN is also less desirable in comparison with GAN, partially due to its requirement for a rather large batch size during the training. In this paper, we propose to improve both the model expressiveness of GMMN and its computational efficiency by introducing adversarial kernel learning techniques, as the replacement of a fixed Gaussian kernel in the original GMMN. The new approach combines the key ideas in both GMMN and GAN, hence we name it MMD GAN. The new distance measure in MMD GAN is a meaningful loss that enjoys the advantage of weak topology and can be optimized via gradient descent with relatively small batch sizes. In our evaluation on multiple benchmark datasets, including MNIST, CIFAR- 10, CelebA and LSUN, the performance of MMD-GAN significantly outperforms GMMN, and is competitive with other representative GAN works.

Motivation & Objective

• Motivate replacing fixed Gaussian kernels in GMMN with adversarially learned kernels to boost expressiveness and efficiency.
• Establish theoretical guarantees: continuity, differentiability, and weak* topology convergence for the kernel-learned MMD objective.
• Develop a practical training algorithm (MMD GAN) with feasible optimization and stability improvements.
• Demonstrate empirical performance improvements over GMMN and competitive results with state-of-the-art GANs on standard datasets.

Proposed method

• Formulate MMD with kernel learning by optimizing over a family of kernels via an injective feature map f_phi, leading to min__theta max__phi M_{k∘f_phi}(P_X, P_theta).
• Use a mixture of RBF kernels with learnable feature transformation to increase testing power.
• Model g_theta (generator) and f_phi (injective feature map / encoder) as neural networks; employ gradient-based optimization with Lipschitz constraints (weight clipping or gradient penalty).
• Regularize with an autoencoder objective to encourage invertibility of f_phi, while noting empirical findings on its necessity.
• Propose feasible set reduction to speed training by constraining the kernel parametric space without changing the objective value.

Experimental results

Research questions

• RQ1 Can adversarially learned kernels in MMD provide stronger two-sample test power than fixed kernels for training generative models?
• RQ2 Does the max over injective feature maps yield a continuous and differentiable objective suitable for gradient-based training?
• RQ3 How does MMD GAN compare to GMMN and to representative GANs (e.g., WGAN) on standard benchmarks in terms of sample quality and diversity?
• RQ4 What is the computational trade-off of MMD GAN relative to GMMN and WGAN, especially with respect to batch size?
• RQ5 Is autoencoder-based regularization necessary for practical training of MMD GAN?

Key findings

• MMD with learned kernels yields a continuous and differentiable objective in generator parameters, enabling gradient descent optimization.
• max_phi M_{f_phi}(P_X, P_theta) converges to P_X in distribution as training progresses (weak* topology property).
• MMD GAN with adversarial kernels achieves sharper, more diverse images than GMMN on MNIST and CIFAR-10, and is competitive with WGAN on several datasets.
• MMD GAN attains meaningful improvements over GMMN while using significantly smaller batch sizes, improving training efficiency.
• Inception scores on CIFAR-10 place MMD GAN above GMMN variants and competitive with many GAN baselines, though DFM shows higher scores.
• WGAN and MMD GAN share a connection: with specific kernel/linearization, MMD GAN aligns with first-order moment matching, while MMD captures higher-order moments via Gaussian kernels.