[Paper Review] Reliable Fidelity and Diversity Metrics for Generative Models

Muhammad Ferjad Naeem, Seong Joon Oh+3 more|arXiv (Cornell University)|Feb 23, 2020

Generative Adversarial Networks and Image SynthesisComputer Science24 references54 citations

TL;DR

The paper introduces density and coverage (D&C) metrics to evaluate fidelity and diversity of generative models, addressing failures of prior precision/recall metrics, and analyzes embedding choices and hyperparameter selection.

ABSTRACT

Devising indicative evaluation metrics for the image generation task remains an open problem. The most widely used metric for measuring the similarity between real and generated images has been the Fréchet Inception Distance (FID) score. Because it does not differentiate the fidelity and diversity aspects of the generated images, recent papers have introduced variants of precision and recall metrics to diagnose those properties separately. In this paper, we show that even the latest version of the precision and recall metrics are not reliable yet. For example, they fail to detect the match between two identical distributions, they are not robust against outliers, and the evaluation hyperparameters are selected arbitrarily. We propose density and coverage metrics that solve the above issues. We analytically and experimentally show that density and coverage provide more interpretable and reliable signals for practitioners than the existing metrics. Code: https://github.com/clovaai/generative-evaluation-prdc.

Motivation & Objective

• Address the instability and hyperparameter arbitrariness of precision and recall metrics for evaluating generative models.
• Propose density and coverage as robust alternatives that separately quantify fidelity and diversity.
• Provide analytic results and empirical evidence showing the advantages of D&C over prior metrics.
• Study embedding choices, including random embeddings, to reduce dataset bias in evaluation.
• Offer practical guidelines for hyperparameter selection and evaluation setup.

Proposed method

• Define density and coverage as neighbourhood-based metrics built from k-nearest neighbours around real samples, aggregating fake sample membership (density) and real sample coverage (coverage).
• Compare D&C to improved precision and recall (P&R) and analyze their robustness to outliers and mode dropping.
• Derive analytic expressions for E[density] = 1 and E[coverage] = 1 - ((N-1)...(N-k))/((M+N-1)...(M+N-k)) under identical real and fake distributions.
• Propose systematic hyperparameter selection aiming for E[coverage] > 0.95 and provide practical defaults (e.g., N=M=10,000, k=5).
• Investigate embedding strategies including ImageNet-pretrained and randomly initialised CNNs, and assess their impact on evaluation across data types (images, audio, etc.).
• Experiment with toy distributions and real datasets (MNIST, FFHQ, CelebA, LSUN, SC09) to illustrate fidelity-diversity diagnostic capabilities.

Experimental results

Research questions

• RQ1 Can density and coverage reliably indicate when real and fake distributions are identical?
• RQ2 Are density and coverage robust to outliers and capable of detecting mode dropping better than prior P&R metrics?
• RQ3 How do embedding choices (pre-trained vs random) affect evaluation outcomes across different domains?
• RQ4 What hyperparameter settings yield stable, distribution-type-agnostic evaluation (e.g., E[coverage] approaching 1) and how should they be chosen in practice?

Key findings

• Density and coverage provide more stable and interpretable signals than precision and recall, especially under outliers and distribution matches.
• Analytic results show E[density] = 1 and E[coverage] approaches 1 as N, M and k grow, enabling principled hyperparameter choices.
• D&C better detect distribution matches and mode dropping, compared with P&R, in toy and real-world experiments.
• Random embeddings can yield more meaningful evaluation when target data substantially diverges from ImageNet statistics.
• Hyperparameters can be systemically chosen to achieve high coverage (e.g., > 0.95), with scalable computation by focusing per-dataset neighbourhoods.