[Paper Review] HellaSwag: Can a Machine Really Finish Your Sentence?

Rowan Zellers, Ari Holtzman+3 more|arXiv (Cornell University)|May 19, 2019

Topic Modeling16 references42 citations

TL;DR

HellaSwag presents a hard adversarially filtered commonsense NLI dataset showing humans are far more accurate than state-of-the-art models like BERT, even after extensive pretraining and fine-tuning.

ABSTRACT

Recent work by Zellers et al. (2018) introduced a new task of commonsense natural language inference: given an event description such as "A woman sits at a piano," a machine must select the most likely followup: "She sets her fingers on the keys." With the introduction of BERT, near human-level performance was reached. Does this mean that machines can perform human level commonsense inference? In this paper, we show that commonsense inference still proves difficult for even state-of-the-art models, by presenting HellaSwag, a new challenge dataset. Though its questions are trivial for humans (>95% accuracy), state-of-the-art models struggle (<48%). We achieve this via Adversarial Filtering (AF), a data collection paradigm wherein a series of discriminators iteratively select an adversarial set of machine-generated wrong answers. AF proves to be surprisingly robust. The key insight is to scale up the length and complexity of the dataset examples towards a critical 'Goldilocks' zone wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models. Our construction of HellaSwag, and its resulting difficulty, sheds light on the inner workings of deep pretrained models. More broadly, it suggests a new path forward for NLP research, in which benchmarks co-evolve with the evolving state-of-the-art in an adversarial way, so as to present ever-harder challenges.

Motivation & Objective

• Motivate evaluation of robust commonsense inference beyond existing benchmarks.
• Expose limitations of pretraining and finetuning in achieving human-like performance.
• Introduce Adversarial Filtering to produce harder, adversarially curated negatives.
• Extend SWAG with WikiHow and ActivityNet contexts to diversify reasoning demands.
• Demonstrate a Goldilocks zone where generated endings are hard for models but easy for humans.

Proposed method

• Use Adversarial Filtering to iteratively curate adversarial, machine-generated negative endings.
• Combine state-of-the-art generators and discriminators to produce hard negatives.
• Expand SWAG with WikiHow and ActivityNet contexts to increase length and diversity of scenarios.
• Evaluate both in-domain and zero-shot generalization, including cross-domain transfer analyses.
• Compare multiple baselines (BERT, GPT, ESIM+ELMo, LSTM, FastText) under a four-way softmax setup.

Experimental results

Research questions

• RQ1 Can current deep pretrained models achieve robust, human-level commonsense reasoning on more challenging, adversarially constructed data?
• RQ2 How do context length, linguistic structure, and domain diversity affect model performance on commonsense NLI?
• RQ3 Do models trained on SWAG transfer to HellaSwag, and vice versa, and what does this say about general commonsense reasoning?
• RQ4 What is the role of pretraining scale and finetuning in solving adversarially constructed commonsense tasks?
• RQ5 How do zero-shot category generalization and domain shifts impact model accuracy on HellaSwag?

Key findings

• Humans achieve around 95% accuracy on HellaSwag, while strong models stay below 50%, even with substantial training data.
• BERT-Large performs best among models (~47.3% overall) but remains far from human performance and struggles with zero-shot categories.
• Adversarial Filtering (AF) creates Endings that fool many models, yet are often nonsensical to humans in the Goldilocks zone of context length (roughly three sentences of context and two generated sentences).
• Transferring SWAG-trained models to HellaSwag yields limited gains (SWAG-trained models score around 34.6% on HellaSwag validation), while HellaSwag-trained models transfer poorly to SWAG (about 69% accuracy on a missing-domain LSMDC task).
• WikiHow generally proves harder for machines than ActivityNet, with humans reaching around 93% accuracy on WikiHow vs. ~45% for Bert-Large; ActivityNet shows higher human-model gaps but lower absolute machine performance.
• The study highlights a need for evolving benchmarks in tandem with model advances, as well as the potential compute bottlenecks required to approach human-level commonsense reasoning.