[Paper Review] An empirical study on evaluation metrics of generative adversarial networks

Qiantong Xu, Gao Huang+5 more|arXiv (Cornell University)|Jun 19, 2018

Generative Adversarial Networks and Image Synthesis27 references221 citations

TL;DR

The paper empirically analyzes popular GAN evaluation metrics and shows that kernel MMD and 1-NN two-sample tests in a learned feature space best satisfy key properties like discriminability, mode sensitivity, and efficiency. It also assesses practical aspects like overfitting detection across GAN models.

ABSTRACT

Evaluating generative adversarial networks (GANs) is inherently challenging. In this paper, we revisit several representative sample-based evaluation metrics for GANs, and address the problem of how to evaluate the evaluation metrics. We start with a few necessary conditions for metrics to produce meaningful scores, such as distinguishing real from generated samples, identifying mode dropping and mode collapsing, and detecting overfitting. With a series of carefully designed experiments, we comprehensively investigate existing sample-based metrics and identify their strengths and limitations in practical settings. Based on these results, we observe that kernel Maximum Mean Discrepancy (MMD) and the 1-Nearest-Neighbor (1-NN) two-sample test seem to satisfy most of the desirable properties, provided that the distances between samples are computed in a suitable feature space. Our experiments also unveil interesting properties about the behavior of several popular GAN models, such as whether they are memorizing training samples, and how far they are from learning the target distribution.

Motivation & Objective

• Clarify desirable properties for GAN evaluation metrics (e.g., discriminability, sensitivity to mode dropping/collapse, and overfitting detection).
• Systematically compare representative sample-based metrics across diverse datasets to identify strengths and limitations.
• Determine which metrics provide reliable guidance for practical GAN development and model selection.

Proposed method

• Review and categorize major sample-based GAN metrics (Inception Score, Mode Score, Kernel MMD, Wasserstein, FID, 1-NN two-sample test).
• Operate metrics in a learned feature space using a pre-trained ResNet-34 to obtain meaningful distances between images.
• Conduct controlled experiments on CelebA and LSUN-bedroom to test discriminability, mode collapse/dropping, robustness to transforms, sample efficiency, and overfitting.
• Evaluate metrics’ sensitivity to real vs. fake mixtures, mode manipulation, and overfitting via holdout validation sets.

Experimental results

Research questions

• RQ1 What are the reasonable behavioral characterizations of existing GAN evaluation metrics?
• RQ2 What are the strengths and limitations of these metrics in practical GAN evaluation?
• RQ3 Which metrics most reliably distinguish real versus generated data and detect issues like mode collapse or overfitting?

Key findings

• Kernel MMD and the 1-NN two-sample test in a convolutional feature space satisfy most desirable properties, including discriminability and efficiency.
• Inception Score and Mode Score can be misleading for datasets very different from ImageNet and fail to detect overfitting.
• Wasserstein distance may require large samples and is computationally intensive, reducing its practical appeal.
• Fréchet Inception Distance (FID) performs robustly and efficiently by modeling moments in feature space.
• The choice of feature space is crucial; convolutional representations (ResNet-based) yield reliable metric behavior over pixel space.
• 1-NN accuracy provides interpretable scores and highlights mode collapse awareness; real vs fake neighbors reveal overfitting tendencies.