[Paper Review] A Practical Guide to Sample-based Statistical Distances for Evaluating Generative Models in Science

Sebastian Bischoff, Alana Darcher+19 more|arXiv (Cornell University)|Mar 19, 2024

Data Analysis with R4 citations

TL;DR

This paper provides a practical, accessible guide to four widely used statistical distances—Sliced-Wasserstein, C2ST, MMD, and FID—for evaluating generative models in scientific domains. It demonstrates through empirical analysis that different distances can yield divergent conclusions on the same data, advocating for multi-distance evaluation to ensure robust model assessment in science.

ABSTRACT

Generative models are invaluable in many fields of science because of their ability to capture high-dimensional and complicated distributions, such as photo-realistic images, protein structures, and connectomes. How do we evaluate the samples these models generate? This work aims to provide an accessible entry point to understanding popular sample-based statistical distances, requiring only foundational knowledge in mathematics and statistics. We focus on four commonly used notions of statistical distances representing different methodologies: Using low-dimensional projections (Sliced-Wasserstein; SW), obtaining a distance using classifiers (Classifier Two-Sample Tests; C2ST), using embeddings through kernels (Maximum Mean Discrepancy; MMD), or neural networks (Fréchet Inception Distance; FID). We highlight the intuition behind each distance and explain their merits, scalability, complexity, and pitfalls. To demonstrate how these distances are used in practice, we evaluate generative models from different scientific domains, namely a model of decision-making and a model generating medical images. We showcase that distinct distances can give different results on similar data. Through this guide, we aim to help researchers to use, interpret, and evaluate statistical distances for generative models in science.

Motivation & Objective

• To provide researchers in science with an accessible, intuitive understanding of key statistical distances used to evaluate generative models.
• To address the challenge of selecting appropriate evaluation metrics when generative models produce complex, high-dimensional data such as images, time series, or neural activity.
• To demonstrate that different statistical distances can yield conflicting evaluations of the same generative model, highlighting the importance of multi-metric assessment.
• To guide researchers in selecting, implementing, and interpreting statistical distances based on data type, domain, and evaluation goals.
• To promote best practices in model evaluation by illustrating the strengths, limitations, and practical considerations of each method using real scientific datasets.

Proposed method

• The authors evaluate four statistical distances: Sliced-Wasserstein (SW), Classifier Two-Sample Test (C2ST), Maximum Mean Discrepancy (MMD), and Fréchet Inception Distance (FID), each representing a distinct methodological approach.
• SW uses low-dimensional projections of high-dimensional data to estimate Wasserstein distance, enabling efficient computation and robustness to high dimensionality.
• C2ST employs a trained classifier to distinguish between real and generated samples, with the test statistic reflecting the separation between distributions.
• MMD computes the distance between embeddings of real and generated data in a reproducing kernel Hilbert space, relying on kernel-based distribution comparison.
• FID uses a pre-trained deep neural network (e.g., Inception) to extract features from images, then computes the Fréchet distance between the mean and covariance of real and generated feature distributions.
• The study applies these metrics to two scientific generative models: one for decision-making processes and another for medical image generation, enabling cross-method comparison.

Figure 1: The need for statistical distances in scientific generative modeling. (a) An example target distribution, $p_{true}(x)$ , and two learned distributions ( $p_{1}(x)$ and $p_{2}(x)$ ) of different models trained to capture $p_{true}(x)$ . All three distributions share the same mean and margi

Experimental results

Research questions

• RQ1 How do different statistical distances—SW, C2ST, MMD, and FID—perform in evaluating generative models across diverse scientific data types?
• RQ2 To what extent do these distances agree in their evaluation of the same generative model, and in what cases do they diverge?
• RQ3 What are the key trade-offs in scalability, complexity, and interpretability among these evaluation methods in real-world scientific applications?
• RQ4 How do domain-specific characteristics—such as temporal structure in time series or perceptual quality in images—affect the suitability of each distance?
• RQ5 Can a single distance reliably assess generative model quality, or is multi-metric evaluation necessary for robust conclusions?

Key findings

• Different statistical distances can yield conflicting evaluations of the same generative model, even when applied to similar data, indicating that no single metric captures all aspects of distributional similarity.
• FID, while widely used, can be sensitive to class imbalance and may not fully reflect perceptual quality, as it relies on potentially biased or non-universal feature extractors.
• C2ST and MMD are effective for general distributional comparison but require careful design of the classifier or kernel to ensure reliable and reproducible results.
• Sliced-Wasserstein offers a strong balance of computational efficiency and robustness to high-dimensional data, making it suitable for complex scientific distributions.
• The choice of feature extractor in FID significantly impacts results, and the lack of standardized, domain-agnostic networks for non-image data (e.g., time series) limits comparability across studies.
• The study demonstrates that combining multiple distances leads to a more comprehensive and reliable assessment of generative model performance than relying on any single metric.

Figure 2: Schematic for the Sliced-Wasserstein distance. (a) Samples from two two-dimensional distributions along with example slices. The “slicing” is done by sampling random directions from the unit sphere and projecting the samples from the higher-dimensional distribution onto that direction. (b)