[Paper Review] DQI: A Guide to Benchmark Evaluation

Swaroop Mishra, Anjana Arunkumar+3 more|arXiv (Cornell University)|Aug 10, 2020

Health Policy Implementation Science23 references4 citations

TL;DR

This paper introduces the Data Quality Index (DQI), a novel, multi-component metric to quantitatively evaluate benchmark quality in NLP by measuring bias and generalizability. DQI outperforms binary and black-box methods by identifying and penalizing spurious correlations across 7 granularities (e.g., words, sentences, trigrams), showing higher DQI values correlate with lower bias and better model robustness on NLI, QA, and RC datasets, including SQuAD 2.0 and Story Cloze.

ABSTRACT

A `state of the art' model A surpasses humans in a benchmark B, but fails on similar benchmarks C, D, and E. What does B have that the other benchmarks do not? Recent research provides the answer: spurious bias. However, developing A to solve benchmarks B through E does not guarantee that it will solve future benchmarks. To progress towards a model that `truly learns' an underlying task, we need to quantify the differences between successive benchmarks, as opposed to existing binary and black-box approaches. We propose a novel approach to solve this underexplored task of quantifying benchmark quality by debuting a data quality metric: DQI.

Motivation & Objective

• To address the limitation of binary and black-box benchmark evaluation methods that fail to quantify bias and generalizability.
• To develop a generic, model-agnostic metric that identifies and penalizes spurious correlations in training data.
• To provide a feedback mechanism for dataset creators to learn from and avoid repeating bias mistakes.
• To reduce resource waste from creating biased datasets by enabling early detection of poor-quality data.
• To support the development of models that 'truly learn' underlying tasks by measuring benchmark quality beyond accuracy.

Proposed method

• DQI is a composite metric with seven components measuring bias and generalizability across multiple granularities: sentences, words, verbs, adjectives, adverbs, nouns, bigrams, and trigrams.
• Each component uses mathematical transformations of data properties—such as sentence similarity (Sim_lm), word similarity (WSim_uv), and premise-hypothesis similarity (ISIM)—with hyperparameters to define thresholds.
• The metric computes terms (T1–T5) based on inter-sample interactions, including overlap in words, maximal word similarity, and sentence length differences.
• DQI is computed per dataset split (e.g., 'Good' vs 'Bad' samples), with higher values indicating lower bias and higher generalizability.
• The method uses empirical formulas derived from a survey of bias types in NLP literature (Mishra et al., 2020b), enabling broad bias coverage without model dependency.
• DQI is validated on NLI, QA, and RC datasets, including SQuAD 2.0 and Story Cloze, using splits labeled as 'Good' or 'Bad' based on model performance.

Experimental results

Research questions

• RQ1 How can we move beyond binary evaluation to quantitatively measure the quality of NLP benchmarks?
• RQ2 What data properties contribute most to spurious bias in NLP datasets?
• RQ3 Can a generic, model-agnostic metric like DQI detect bias more effectively than existing binary or black-box approaches?
• RQ4 How does DQI correlate with model generalization and robustness across different NLP tasks?
• RQ5 To what extent can DQI guide dataset creators in identifying and removing biased samples?

Key findings

• On SQuAD 2.0, DQI_c6 values were 75,918.28 for 'Good' samples and 105,949.34 for 'Bad' samples, indicating higher bias in the latter.
• For Story Cloze, DQI_c6 reached 1.01×10^17 for both 'Good' and 'Bad' splits, suggesting high bias or saturation in the metric under certain conditions.
• In MNLI, DQI_c7 showed low values (0.0004–0.0011) for 'Good' and 'Bad' splits, indicating poor separation between high- and low-quality samples.
• For SNLI, DQI_c7 values were 0.0004–0.0005 for 'Good' and 0.0009–0.0011 for 'Bad', showing a measurable difference in quality between splits.
• DQI components revealed that word overlap and sentence similarity were key contributors to bias, with 'Bad' splits showing higher similarity and lower diversity.
• The metric demonstrated that higher DQI values correlate with lower model overfitting and better generalization, especially in NLI and QA benchmarks.