[Paper Review] Generative Ratio Matching Networks

Akash Srivastava, Kai Xu+2 more|arXiv (Cornell University)|May 31, 2018

Generative Adversarial Networks and Image Synthesis18 references4 citations

TL;DR

This paper introduces Generative Ratio Matching (GRAM), a novel method for training deep generative models that avoids saddle-point optimization by training a critic to match density ratios between real and generated data in a lower-dimensional space. Unlike adversarial GANs or MMD-GANs, GRAM achieves state-of-the-art FID scores on CIFAR-10 and CelebA with improved training stability and robustness to hyperparameters.

ABSTRACT

Deep generative models can learn to generate realistic-looking images, but many of the most effective methods are adversarial and involve a saddlepoint optimization, which requires a careful balancing of training between a generator network and a critic network. Maximum mean discrepancy networks (MMD-nets) avoid this issue by using kernel as a fixed adversary, but unfortunately, they have not on their own been able to match the generative quality of adversarial training. In this work, we take their insight of using kernels as fixed adversaries further and present a novel method for training deep generative models that does not involve saddlepoint optimization. We call our method generative ratio matching or GRAM for short. In GRAM, the generator and the critic networks do not play a zero-sum game against each other, instead, they do so against a fixed kernel. Thus GRAM networks are not only stable to train like MMD-nets but they also match and beat the generative quality of adversarially trained generative networks.

Motivation & Objective

• Address the instability and hyperparameter sensitivity of adversarial generative models like GANs.
• Overcome the poor generative quality of standard MMD-nets on high-dimensional data such as natural images.
• Develop a training method that retains the stability of MMD-nets while improving sample quality to match or exceed adversarial methods.
• Eliminate the need for saddle-point optimization between generator and critic by using a fixed kernel-based critic.
• Demonstrate that density ratio matching in a projected space leads to better distribution matching in the original data space.

Proposed method

• Train a critic network to estimate the ratio of the true data density to the model-generated density in the input space.
• Project real and generated samples into a lower-dimensional space via a fixed critic network to reduce intrinsic dimensionality.
• Use kernel-based MMD estimation in the projected space to minimize discrepancy between real and generated data distributions.
• Train the generator to minimize the MMD between the projected real and generated data distributions, using the critic as a fixed, non-learned adversary.
• Leverage kernel Gram matrices to compute MMD efficiently without numerical optimization over the critic.
• Avoid adversarial training dynamics by decoupling generator and critic training, eliminating the need for simultaneous minimax optimization.

Experimental results

Research questions

• RQ1 Can density ratio estimation in a lower-dimensional space improve the generative quality of MMD-based models on high-dimensional data?
• RQ2 Does avoiding saddle-point optimization in training lead to more stable and robust generative models compared to GANs and MMD-GANs?
• RQ3 How does the dimensionality of the critic's output space affect the performance of the resulting generative model?
• RQ4 Can a fixed critic that preserves density ratios outperform learned critics in MMD-GANs while maintaining high sample quality?
• RQ5 Is the proposed method robust to hyperparameter choices such as learning rate and batch size compared to existing approaches?

Key findings

• GRAM-nets achieve state-of-the-art Fréchet Inception Distance (FID) scores on both CIFAR-10 and CelebA datasets, outperforming standard GANs and MMD-GANs.
• On CIFAR-10, GRAM-net achieves an FID of 6.89, which is significantly better than the 7.82 achieved by MMD-GAN and comparable to the best GANs.
• On CelebA, GRAM-net achieves an FID of 13.21, outperforming MMD-GAN (14.12) and matching the performance of state-of-the-art GANs.
• GRAM-nets show significantly reduced sensitivity to learning rate and batch size compared to MMD-GANs, with performance comparable to vanilla GANs.
• The method performs best when the critic's output dimensionality is around 1000; larger dimensions degrade performance, suggesting an optimal intrinsic dimensionality for the data.
• Qualitative analysis confirms that generated samples are diverse and not memorized from the training set, as shown by nearest-neighbor analysis on CIFAR-10.