[Paper Review] Towards GAN Benchmarks Which Require Generalization

Ishaan Gulrajani, Colin Raffel+1 more|arXiv (Cornell University)|Jan 10, 2020

Model-Driven Software Engineering TechniquesComputer Science33 citations

TL;DR

The paper argues that common GAN evaluation metrics can be gamed by memorization and proposes neural network divergences (NNDs) as sample-based benchmarks that require genuine generalization; it demonstrates a CNN-based divergence that penalizes memorization and correlates with perceptual quality.

ABSTRACT

For many evaluation metrics commonly used as benchmarks for unconditional image generation, trivially memorizing the training set attains a better score than models which are considered state-of-the-art; we consider this problematic. We clarify a necessary condition for an evaluation metric not to behave this way: estimating the function must require a large sample from the model. In search of such a metric, we turn to neural network divergences (NNDs), which are defined in terms of a neural network trained to distinguish between distributions. The resulting benchmarks cannot be "won" by training set memorization, while still being perceptually correlated and computable only from samples. We survey past work on using NNDs for evaluation and implement an example black-box metric based on these ideas. Through experimental validation we show that it can effectively measure diversity, sample quality, and generalization.

Motivation & Objective

• Define a meaningful, sample-based benchmark for unconditional image generation that requires generalization beyond memorization.
• Clarify how evaluation metrics should incentivize generalization rather than training-set memorization.
• Investigate neural network divergences (NNDs) as such benchmarks and assess their properties.

Proposed method

• Frame evaluation metrics as divergences between distributions to be estimated from finite samples.
• Propose a baseline where models must outperform training-set memorization (Definition 1).
• Introduce CNN divergence (D_CNN) as a practical NND using a CNN critic and WGAN-GP objectives, plus an experimental protocol to evaluate generalization.
• Evaluate how many training samples are needed for memorization to beat GANs across metrics (Table 2).
• Compare D_CNN to Inception Score (IS) and Fréchet Inception Distance (FID) on CIFAR-10.

Experimental results

Research questions

• RQ1 Can a sample-based metric be designed so memorization cannot trivially beat a model, thus enforcing generalization?
• RQ2 Do neural network divergences effectively detect memorization vs. genuine generalization in GANs?
• RQ3 How do perceptually aligned NNDs (e.g., CNN-based) compare to IS and FID in correlating with sample diversity and quality?

Key findings

• IS and FID can favor memorization over learned generalization, while CNN divergence prefers a model that generalizes beyond the training set.
• CNN divergence detects overfitting and measures diversity more effectively than IS/FID in the presented experiments.
• A CNN divergence trained with a WGAN-GP setup can outperform memorization on CIFAR-10 when evaluated with D_CNN on test samples (and correlates with broader generalization).
• Increasing the required sample size for the critic raises the threshold at which memorization can beat the model, indicating sensitivity to diversity.
• The CNN divergence values decrease during training and reveal a gap between training and test divergences, underscoring the importance of generalization in evaluation.
• Bias from small test sets can affect estimates, but the trend across larger vs. smaller test sets remains correlated.