[Paper Review] MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers

Krishna Pillutla, Swabha Swayamdipta+5 more|arXiv (Cornell University)|Feb 2, 2021

Topic Modeling91 citations

TL;DR

MAUVE introduces a divergence-frontier based metric to quantify how close neural text distributions are to human text by comparing model and human text distributions in a quantized embedding space.

ABSTRACT

As major progress is made in open-ended text generation, measuring how close machine-generated text is to human language remains a critical open problem. We introduce MAUVE, a comparison measure for open-ended text generation, which directly compares the learnt distribution from a text generation model to the distribution of human-written text using divergence frontiers. MAUVE scales up to modern text generation models by computing information divergences in a quantized embedding space. Through an extensive empirical study on three open-ended generation tasks, we find that MAUVE identifies known properties of generated text, scales naturally with model size, and correlates with human judgments, with fewer restrictions than existing distributional evaluation metrics.

Motivation & Objective

• Motivate a principled, scalable measure for open-ended text generation that accounts for both degenerate machine outputs and limited coverage of human text.
• Formalize the gap between model and human distributions using soft Type I and Type II errors via divergence frontiers.
• Provide an efficient estimation framework by embedding and quantizing text distributions to compute KL divergences in a low-dimensional space.
• Demonstrate that Mauve correlates with human judgments and is robust to embedding/quantization choices.

Proposed method

• Define the divergence curve C(P,Q) as the set of pairs (KL(P|Rλ), KL(Q|Rλ)) for Rλ = λP + (1−λ)Q with λ ∈ (0,1).
• Approximate the unknown distributions P and Q with samples from human and machine text, embedded via an external model M, yielding vectors in R^d.
• Quantize the embedded samples (e.g., k-means) to form discrete distributions ŜP and ŜQ over k bins.
• Compute the area under the divergence curve Mauve(P,Q) as a scalar summary of the trade-off between Type I and Type II errors.
• Provide a practical estimation workflow that uses MC sampling and a chosen scaling constant c to produce interpretable Mauve values.
• Show robustness of Mauve to embedding choices and quantization methods.

Experimental results

Research questions

• RQ1 How close is neural text distribution Q to human text distribution P when evaluated across a spectrum of mixture proportions λ?
• RQ2 Can Mauve identify known properties of generated text such as effects of length, decoding algorithm, and model size?
• RQ3 Is Mauve robust to different embedding and quantization strategies while correlating with human judgments?
• RQ4 Does Mauve offer a more faithful correlation with human quality judgments than existing automatic metrics?

Key findings

• Mauve correlates highly with human judgments on perceived quality, being higher for more human-like and sensible outputs compared to alternative metrics.
• Mauve captures the expected quality relationships across decoding algorithms (greedy < ancestral < nucleus) and detects degeneracy in adversarial/beam-search decodings.
• Mauve increases with larger model size and aligns with human quality assessments, whereas some traditional metrics like generation perplexity do not.
• Mauve remains robust to the choice of embedding (GPT-2 Large vs RoBERTa Large) and to different quantization strategies (k-means, DRMM, lattice).
• The approach provides a scalable, domain-agnostic measure for open-ended text generation that summarizes the distributional gap between machine and human text.