[Paper Review] Assessing Generative Models via Precision and Recall

Mehdi S. M. Sajjadi, Olivier Bachem+3 more|arXiv (Cornell University)|May 31, 2018

Generative Adversarial Networks and Image SynthesisComputer Science173 citations

TL;DR

Introduces a distribution-level precision and recall framework (PRD) to evaluate generative models, enabling separation of sample quality from distribution coverage and providing an efficient computation method from samples.

ABSTRACT

Recent advances in generative modeling have led to an increased interest in the study of statistical divergences as means of model comparison. Commonly used evaluation methods, such as the Frechet Inception Distance (FID), correlate well with the perceived quality of samples and are sensitive to mode dropping. However, these metrics are unable to distinguish between different failure cases since they only yield one-dimensional scores. We propose a novel definition of precision and recall for distributions which disentangles the divergence into two separate dimensions. The proposed notion is intuitive, retains desirable properties, and naturally leads to an efficient algorithm that can be used to evaluate generative models. We relate this notion to total variation as well as to recent evaluation metrics such as Inception Score and FID. To demonstrate the practical utility of the proposed approach we perform an empirical study on several variants of Generative Adversarial Networks and Variational Autoencoders. In an extensive set of experiments we show that the proposed metric is able to disentangle the quality of generated samples from the coverage of the target distribution.

Motivation & Objective

• Motivate the need for evaluation metrics that distinguish sample quality from coverage of the target distribution.
• Define a theoretically sound precision and recall notion for distributions that decomposes divergences.
• Develop an efficient algorithm to compute PRD curves from samples.
• Relate PRD to established metrics and demonstrate practical utility on image and text data.
• Provide empirical insight into GANs and VAEs using PRD to diagnose mode dropping and mode inventing.

Proposed method

• Define P and Q as distributions on a finite space and decompose into shared and non-shared components.
• Introduce a trade-off parameterized PRD framework using a common component μ to capture shared structure and derive a formal precision-recall set PRD(Q,P).
• Prove foundational properties of PRD, including duality and monotonicity, and connect to total variation.
• Provide an algorithm to compute PRD via alpha(lambda) and beta(lambda) sums and an equiangular grid over lambda.
• Apply PRD to deep generative models by embedding samples in a feature space (Inception Pool3) and clustering to reduce to a 1D comparison.
• Demonstrate computation and interpretation of PRD on MNIST, Fashion-MNIST, CIFAR-10, CelebA, and text data.

Experimental results

Research questions

• RQ1 How can precision and recall be defined for distributions to reveal different failure modes of generative models?
• RQ2 How does the proposed PRD framework relate to existing metrics like total variation, IS, and FID?
• RQ3 Can PRD disentangle mode dropping from mode invention in GANs and VAEs?
• RQ4 Is PRD practical to compute from samples and meaningful when applied to image and text data?
• RQ5 What empirical insights about GANs vs VAEs does PRD yield regarding sample quality and diversity?

Key findings

• PRD provides a two-dimensional view that separates sample quality (precision) from distribution coverage (recall).
• The PRD framework generalizes total variation and connects to FID and Inception Score.
• PRD curves reveal mode dropping vs mode invention in GANs and VAEs, clarifying why FID/IS may be inconclusive.
• Empirical results show VAEs tend to higher recall but lower precision, whereas GANs often yield higher precision but lower recall.
• The method works across image and text data by embedding via a pre-trained classifier and clustering in feature space.
• PRD offers a family of summaries through Fβ by trading precision vs recall, aiding model comparison.