[Paper Review] Adversarial NLI: A New Benchmark for Natural Language Understanding

Yixin Nie, Adina Williams+4 more|arXiv (Cornell University)|Oct 31, 2019

Topic Modeling63 references66 citations

TL;DR

Introduces ANLI, a large adversarially collected NLI benchmark built with a human-and-model-in-the-loop procedure across three rounds, yielding harder test sets and state-of-the-art results on SNLI and MNLI when trained on ANLI data.

ABSTRACT

We introduce a new large-scale NLI benchmark dataset, collected via an iterative, adversarial human-and-model-in-the-loop procedure. We show that training models on this new dataset leads to state-of-the-art performance on a variety of popular NLI benchmarks, while posing a more difficult challenge with its new test set. Our analysis sheds light on the shortcomings of current state-of-the-art models, and shows that non-expert annotators are successful at finding their weaknesses. The data collection method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate.

Motivation & Objective

• Motivate the creation of a long-lasting, harder NLI benchmark that reveals model weaknesses beyond existing datasets.
• Propose HAMLET (Human-And-Model-in-the-Loop Enabled Training) to iteratively collect adversarial examples from humans targeting model errors.
• Show that training on ANLI improves robustness and achieves state-of-the-art results on SNLI and MNLI.
• Analyze the types of inferences that fool current models and assess annotator biases and dataset dynamics.

Proposed method

• Iteratively collect NLI data through three rounds with increasing difficulty using HAMLET, where humans write hypotheses intended to mislead the current models.
• Verify writer-correctness with multiple human validators to ensure target labels are correct for training and test splits.
• Train new models per round (and ensembles in later rounds) on the accumulated data and generate new test sets solely from verified errors.
• Use longer, multi-sentence contexts drawn from diverse sources (Wikipedia, HotpotQA, News, fiction, legal/open-domain transcripts) to increase inference difficulty.
• Include annotator-provided explanations and per-example inference-type labels to enable finer-grained analysis of model weaknesses.

Experimental results

Research questions

• RQ1 Can a human-and-model-in-the-loop data collection create a moving, harder NLI benchmark that modestly resists current state-of-the-art models?
• RQ2 Does training on adversarial ANLI data improve performance and robustness on standard NLI benchmarks (SNLI/MNLI) and stress tests?
• RQ3 What are the predominant inference types that fool modern NLI models, and how do model weaknesses evolve across rounds?
• RQ4 Do exclusive test annotators introduce significant stylistic biases, and how does this affect generalization?
• RQ5 Is adversarial data more data-efficient than traditional NLI data for improving robustness?

Key findings

• ANLI's three-round procedure yields a growingly difficult test set with decreasing model error rates across rounds as models become more robust.
• RoBERTa finetuned on ANLI achieves state-of-the-art on SNLI (92.9) and MNLI (91.0/90.7 for matched/mismatched), outperforming prior baselines on these benchmarks.
• Training on ANLI data improves robustness across SNLI, MNLI, and stress tests, indicating data efficiency of adversarial examples compared to standard data.
• Hypothesis-only models perform poorly on ANLI, especially in later rounds, suggesting genuine reliance on reasoning beyond superficial hypothesis cues.
• Exclusive test annotators show negligible difference from general test sets, indicating limited overfitting to annotator style.