[Paper Review] Understanding Dataset Difficulty with $\mathcal{V}$-Usable Information

Kawin Ethayarajh, Yejin Choi+1 more|arXiv (Cornell University)|Oct 16, 2021

Topic Modeling4 citations

TL;DR

This paper introduces a novel information-theoretic framework using $Π$-usable information to quantify dataset difficulty for a given model $Π$, with pointwise $Π$-information (pvi) enabling instance-level difficulty estimation. It reveals that lower $Π$-usable information correlates with higher model difficulty and identifies annotation artifacts in major NLP benchmarks, such as mislabeled instances and over-reliance on minimal lexical cues.

ABSTRACT

Estimating the difficulty of a dataset typically involves comparing state-of-the-art models to humans; the bigger the performance gap, the harder the dataset is said to be. However, this comparison provides little understanding of how difficult each instance in a given distribution is, or what attributes make the dataset difficult for a given model. To address these questions, we frame dataset difficulty -- w.r.t. a model $\mathcal{V}$ -- as the lack of $\mathcal{V}$-$ extit{usable information}$ (Xu et al., 2019), where a lower value indicates a more difficult dataset for $\mathcal{V}$. We further introduce $ extit{pointwise $\mathcal{V}$-information}$ (PVI) for measuring the difficulty of individual instances w.r.t. a given distribution. While standard evaluation metrics typically only compare different models for the same dataset, $\mathcal{V}$-$ extit{usable information}$ and PVI also permit the converse: for a given model $\mathcal{V}$, we can compare different datasets, as well as different instances/slices of the same dataset. Furthermore, our framework allows for the interpretability of different input attributes via transformations of the input, which we use to discover annotation artefacts in widely-used NLP benchmarks.

Motivation & Objective

• To address the lack of interpretability in standard dataset difficulty evaluation, which relies only on aggregate model-human performance gaps.
• To develop a model-specific metric for dataset difficulty that quantifies how much information a model can actually use from inputs.
• To enable fine-grained analysis of individual instances and data slices to uncover hidden biases and annotation artifacts in widely used NLP benchmarks.
• To provide a stable, interpretable, and computationally grounded alternative to uncertainty-based or IRT-based difficulty estimation.

Proposed method

• Formalize dataset difficulty as the lack of $Π$-usable information, defined via predictive $Π$-information ($I_{Π}(X\to Y)$), which measures information extractable under model constraints.
• Introduce pointwise $Π$-information (pvi) as a per-instance measure of difficulty, derived from the model’s predictive capacity on individual inputs.
• Use input transformations (e.g., word shuffling, synonym replacement) to isolate the contribution of specific input attributes to $Π$-usable information.
• Compare pvi values across models, training seeds, and time steps to validate stability and correlation with human judgments.
• Apply the framework to detect annotation artifacts by measuring changes in $Π$-usable information after manipulating input attributes.
• Use the framework to compare datasets and slices under the same model, enabling cross-dataset difficulty analysis.

Experimental results

Research questions

• RQ1 How can dataset difficulty be formally quantified in a way that reflects the actual usability of information by a specific model $Π$?
• RQ2 To what extent does pointwise $Π$-information (pvi) correlate with human-annotated difficulty and model prediction accuracy?
• RQ3 Which input attributes contribute most to the usability of information in a given dataset for a given model?
• RQ4 How can systematic annotation artifacts in NLP benchmarks be detected using $Π$-usable information?

Key findings

• The Stanford NLI (SNLI) dataset contains significantly more BERT-usable information than MultiNLI and CoLA, explaining BERT’s superior performance on SNLI.
• pvi values are highly correlated (Pearson $r \geq 0.75$) across different models, training seeds, and training times, indicating stability and reliability.
• Instances with lower pvi values show lower annotator agreement and lower model prediction accuracy, validating pvi as a proxy for difficulty.
• In SNLI, word ordering has minimal impact on BERT-usable information, suggesting models may rely on shallow cues rather than semantic structure.
• In CoLA, some of the most difficult instances are mislabeled, as indicated by low pvi and high model uncertainty.
• In the hate speech detection dataset, just 50 potentially offensive words account for most of the BERT-usable information, indicating a risk of undetected bias.