[Paper Review] MAUVE Scores for Generative Models: Theory and Practice

Krishna Pillutla, Lang Liu+7 more|arXiv (Cornell University)|Dec 30, 2022

Generative Adversarial Networks and Image Synthesis5 citations

TL;DR

This paper introduces Mauve, a family of statistical divergence scores for evaluating generative models by summarizing f-divergence frontiers between real and generated data distributions. It proposes three estimation methods—vector quantization, nearest neighbors, and classification—providing theoretical bounds and demonstrating strong correlation with human judgments across text and image generation tasks.

ABSTRACT

Generative artificial intelligence has made significant strides, producing text indistinguishable from human prose and remarkably photorealistic images. Automatically measuring how close the generated data distribution is to the target distribution is central to diagnosing existing models and developing better ones. We present MAUVE, a family of comparison measures between pairs of distributions such as those encountered in the generative modeling of text or images. These scores are statistical summaries of divergence frontiers capturing two types of errors in generative modeling. We explore three approaches to statistically estimate these scores: vector quantization, non-parametric estimation, and classifier-based estimation. We provide statistical bounds for the vector quantization approach. Empirically, we find that the proposed scores paired with a range of $f$-divergences and statistical estimation methods can quantify the gaps between the distributions of human-written text and those of modern neural language models by correlating with human judgments and identifying known properties of the generated texts. We demonstrate in the vision domain that MAUVE can identify known properties of generated images on par with or better than existing metrics. In conclusion, we present practical recommendations for using MAUVE effectively with language and image modalities.

Motivation & Objective

• To address the challenge of evaluating generative models by quantifying distributional gaps between real and generated data.
• To formalize a principled approach for measuring two key errors in generative modeling: out-of-distribution samples and missing real data modes.
• To develop a family of interpretable, scalable metrics—Mauve scores—that summarize divergence frontiers for text and image generation.
• To provide statistical estimation methods with theoretical guarantees and practical recommendations for deployment in NLP and vision.

Proposed method

• Define Mauve scores as scalar summaries of f-divergence frontiers, capturing trade-offs between two types of modeling errors.
• Use vector quantization to discretize latent representations and estimate divergence frontiers with statistical bounds.
• Implement non-parametric nearest-neighbor estimation to approximate f-divergences using local density ratios.
• Apply classifier-based estimation via binary classification between real and generated samples to estimate divergence metrics.
• Derive theoretical statistical error bounds for vector quantization and smoothing-based approaches.
• Integrate Mauve scores with various f-divergences (e.g., KL, JS) and embedding spaces to assess model quality across modalities.

Experimental results

Research questions

• RQ1 How can divergence frontiers be effectively summarized into a single, interpretable metric for evaluating generative models?
• RQ2 What are the statistical properties and error bounds of different estimation methods (vector quantization, nearest neighbors, classification) for Mauve scores?
• RQ3 To what extent do Mauve scores correlate with human judgments of text quality and realism?
• RQ4 How do Mauve scores detect known failure modes in text and image generation, such as repetition or distributional shift?
• RQ5 What are the optimal configurations and practical recommendations for deploying Mauve in real-world model evaluation pipelines?

Key findings

• Mauve scores show strong correlation with human judgments across multiple evaluation tasks, particularly in ranking model quality by size and decoding strategy.
• The vector quantization method achieves tight statistical error bounds, with theoretical guarantees on convergence under mild regularity conditions.
• GPT-2 small outperforms GPT-2 medium in pure sampling settings, a nuance captured by Mauve but not always reflected in standard metrics.
• Mauve identifies distributional gaps in image generation models with performance on par with or better than existing metrics like FID and LPIPS.
• The method effectively detects mode collapse and distributional shift in both text and image generation, especially when combined with f-divergences.
• Classifier-based estimation provides a robust and scalable alternative to non-parametric methods, particularly in high-dimensional latent spaces.