[Paper Review] How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models

Ahmed M. Alaa, Boris van Breugel+2 more|arXiv (Cornell University)|Feb 17, 2021

Scientific Computing and Data ManagementDecision Sciences34 citations

TL;DR

Introduces alpha-Precision, beta-Recall, and Authenticity as a 3D, sample-level metric to evaluate fidelity, diversity, and generalization of generative models across domains, plus a model-auditing use case.

ABSTRACT

Devising domain- and model-agnostic evaluation metrics for generative models is an important and as yet unresolved problem. Most existing metrics, which were tailored solely to the image synthesis setup, exhibit a limited capacity for diagnosing the different modes of failure of generative models across broader application domains. In this paper, we introduce a 3-dimensional evaluation metric, ($\\alpha$-Precision, $\\beta$-Recall, Authenticity), that characterizes the fidelity, diversity and generalization performance of any generative model in a domain-agnostic fashion. Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample- and distribution-level diagnoses of model fidelity and diversity. We introduce generalization as an additional, independent dimension (to the fidelity-diversity trade-off) that quantifies the extent to which a model copies training data -- a crucial performance indicator when modeling sensitive data with requirements on privacy. The three metric components correspond to (interpretable) probabilistic quantities, and are estimated via sample-level binary classification. The sample-level nature of our metric inspires a novel use case which we call model auditing, wherein we judge the quality of individual samples generated by a (black-box) model, discarding low-quality samples and hence improving the overall model performance in a post-hoc manner.

Motivation & Objective

• Provide domain- and model-agnostic metrics for evaluating generative models beyond image-specific measures.
• Quantify three qualities of synthetic data: fidelity, diversity, and generalization.
• Enable sample-level diagnostics and a novel model-auditing use case to post-hoc improve synthetic data quality.
• Offer a practical embedding and hypothesis-testing framework to estimate the metrics across domains.

Proposed method

• Define alpha-Precision as the probability a synthetic sample lies in the alpha-support of real data.
• Define beta-Recall as the probability a real sample lies in the beta-support of synthetic data.
• Define Authenticity as the probability a synthetic sample is not a memorized training copy, modeled via a mixture with a noise component.
• Embed real and synthetic data via a model-driven evaluation embedding that maps to a hypersphere to estimate alpha- and beta-subssets.
• Estimate sample-level scores with three classifiers for precision, recall, and authenticity based on the embedded representations.
• Provide a post-hoc auditing workflow that discards low-quality samples to curate higher-quality synthetic data.

Experimental results

Research questions

• RQ1 Can a domain- and model-agnostic, sample-level metric simultaneously capture fidelity, diversity, and generalization for generative models?
• RQ2 Do alpha-Precision, beta-Recall, and Authenticity provide meaningful, interpretable diagnostics beyond traditional distribution-level metrics?
• RQ3 Can model auditing using these metrics improve downstream predictive performance when trained on synthetic data?
• RQ4 How do the proposed metrics perform in real-world sensitive-data scenarios (e.g., clinical COVID-19 data) and in multimodal simple benchmarks like MNIST?

Key findings

• alpha-Precision and beta-Recall curves reveal fidelity and diversity across density levels, addressing failures like mode collapse and mode invention.
• Integrated metrics IP_alpha and IR_beta correlate with ground-truth model quality and can outperform standard P1/R1 and other distribution-based metrics in ranking generative models.
• Authenticity captures generalization by distinguishing novel samples from memorized training data, enabling privacy-aware evaluation.
• Model auditing using sample-level scores improves downstream predictive performance, demonstrated by improved AUC in COVID-19 mortality prediction after auditing synthetic data.
• The framework detects mode dropping on MNIST by showing declines in IR_beta while IP_alpha remains robust, highlighting its ability to diagnose specific failure modes.