[Paper Review] Evaluating NLP Models via Contrast Sets

Matt Gardner, Yoav Artzi+24 more|arXiv (Cornell University)|Apr 6, 2020

Topic Modeling64 references61 citations

TL;DR

This paper introduces contrast sets—manually created perturbations of test instances that change the gold label while preserving meaning—to expose systematic gaps in NLP datasets. By evaluating models on these contrast sets, the authors show significant performance drops (up to 25%) compared to original test sets, revealing that models often rely on simple, non-robust decision rules rather than true linguistic understanding.

ABSTRACT

Standard test sets for supervised learning evaluate in-distribution generalization. Unfortunately, when a dataset has systematic gaps (e.g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities. We propose a new annotation paradigm for NLP that helps to close systematic gaps in the test data. In particular, after a dataset is constructed, we recommend that the dataset authors manually perturb the test instances in small but meaningful ways that (typically) change the gold label, creating contrast sets. Contrast sets provide a local view of a model's decision boundary, which can be used to more accurately evaluate a model's true linguistic capabilities. We demonstrate the efficacy of contrast sets by creating them for 10 diverse NLP datasets (e.g., DROP reading comprehension, UD parsing, IMDb sentiment analysis). Although our contrast sets are not explicitly adversarial, model performance is significantly lower on them than on the original test sets---up to 25\% in some cases. We release our contrast sets as new evaluation benchmarks and encourage future dataset construction efforts to follow similar annotation processes.

Motivation & Objective

• To address the problem of misleading model evaluations due to systematic gaps in standard NLP test sets.
• To propose a new annotation paradigm that identifies and corrects these gaps by creating contrast sets.
• To improve the accuracy of model evaluation by measuring robustness to small, meaningful perturbations.
• To demonstrate the effectiveness of contrast sets across 10 diverse NLP datasets, showing that models perform significantly worse on contrast sets than on original test sets.

Proposed method

• After dataset construction, authors manually perturb test instances in small, meaningful ways that typically change the gold label.
• Perturbations are designed to be minimal yet semantically significant, preserving the original instance's context while altering the correct label.
• The resulting contrast sets provide a local view of a model’s decision boundary, revealing reliance on spurious patterns.
• Contrast sets are created for 10 diverse NLP datasets, including DROP, IMDb, and dependency parsing tasks.
• Models are evaluated on both original test sets and contrast sets to measure performance degradation.
• The approach is non-adversarial but exposes model brittleness by highlighting failure cases due to dataset artifacts.

Experimental results

Research questions

• RQ1 To what extent do standard NLP test sets fail to evaluate a model’s true linguistic capabilities due to systematic gaps?
• RQ2 How effective are contrast sets in exposing models’ reliance on spurious patterns rather than robust linguistic understanding?
• RQ3 What is the magnitude of performance drop when models are evaluated on contrast sets compared to original test sets?
• RQ4 Can contrast sets be systematically constructed across diverse NLP tasks to serve as a reliable evaluation benchmark?

Key findings

• Model performance on contrast sets dropped significantly compared to original test sets, with reductions of up to 25% on some datasets.
• The performance drop indicates that models often exploit dataset artifacts rather than learning generalizable linguistic patterns.
• Contrast sets successfully revealed systematic gaps in 10 diverse NLP datasets, including reading comprehension, sentiment analysis, and parsing.
• The proposed method does not require adversarial training or complex optimization, yet it effectively exposes model weaknesses.
• The contrast sets are released as new evaluation benchmarks to encourage more robust dataset construction in future NLP research.
• The approach demonstrates that even non-adversarial perturbations can expose critical flaws in model generalization.