[Paper Review] SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference

Rowan Zellers, Yonatan Bisk+2 more|arXiv (Cornell University)|Aug 15, 2018

Topic Modeling4 citations

TL;DR

This paper introduces Swag, a large-scale adversarial dataset of 113,000 grounded commonsense inference examples derived from video captions, using Adversarial Filtering to eliminate annotation artifacts. The method combines language model–generated counterfactuals with a committee of stylistic classifiers to create de-biased multiple-choice questions, achieving 88% human accuracy but less than 60% for state-of-the-art models, highlighting a significant challenge for NLP systems.

ABSTRACT

Given a partial description like "she opened the hood of the car," humans can reason about the situation and anticipate what might come next ("then, she examined the engine"). In this paper, we introduce the task of grounded commonsense inference, unifying natural language inference and commonsense reasoning. We present SWAG, a new dataset with 113k multiple choice questions about a rich spectrum of grounded situations. To address the recurring challenges of the annotation artifacts and human biases found in many existing datasets, we propose Adversarial Filtering (AF), a novel procedure that constructs a de-biased dataset by iteratively training an ensemble of stylistic classifiers, and using them to filter the data. To account for the aggressive adversarial filtering, we use state-of-the-art language models to massively oversample a diverse set of potential counterfactuals. Empirical results demonstrate that while humans can solve the resulting inference problems with high accuracy (88%), various competitive models struggle on our task. We provide comprehensive analysis that indicates significant opportunities for future research.

Motivation & Objective

• To address the pervasive issue of annotation artifacts and human biases in NLP datasets that mislead models into relying on stylistic cues rather than true reasoning.
• To develop a scalable, automated method for constructing high-quality, de-biased datasets for grounded commonsense inference.
• To create a benchmark that challenges state-of-the-art models by requiring physical and situational reasoning beyond linguistic entailment.
• To demonstrate that current models struggle significantly on grounded commonsense tasks despite high human performance.

Proposed method

• Collect 113,000 video caption pairs from ActivityNet and LSMDC, forming context-verb phrase pairs for multiple-choice inference.
• Use a fine-tuned state-of-the-art language model to massively oversample diverse, plausible counterfactual (negative) verb phrase endings.
• Apply Adversarial Filtering (AF): train an ensemble of stylistic classifiers to identify and filter out endings with biased linguistic patterns.
• Iteratively refine the dataset by retraining the classifier on filtered data to eliminate stylistic artifacts while preserving semantic plausibility.
• Validate the final filtered counterfactuals through crowd-sourcing to ensure quality and diversity.
• Construct a balanced dataset with four options per context: one correct, three de-biased, adversarially filtered counterfactuals.

Experimental results

Research questions

• RQ1 Can adversarial filtering effectively reduce annotation artifacts in large-scale commonsense reasoning datasets?
• RQ2 To what extent does the Swag dataset expose the limitations of current NLP models in grounded commonsense reasoning?
• RQ3 How does human performance on grounded commonsense inference compare to state-of-the-art models?
• RQ4 Does the use of language model–generated counterfactuals improve the robustness and diversity of negative examples in inference datasets?

Key findings

• Humans achieve 88% accuracy on the Swag dataset, indicating that the task is intuitive for humans despite its complexity.
• State-of-the-art NLI models achieve less than 60% accuracy on Swag, demonstrating a significant performance gap compared to human performance.
• Adversarial Filtering successfully reduces stylistic artifacts, as evidenced by the fact that multiple classifiers cannot reliably distinguish correct from incorrect options.
• The dataset exhibits high diversity in verbs and topics, with no single verb dominating the distribution, suggesting broad coverage of real-world situations.
• Even the best-performing model (ESIM+ELMo) frequently selects incorrect answers with high confidence, indicating a failure to reason about physical plausibility.
• The cumulative distribution of verbs in Swag is less skewed than in MultiNLI, suggesting broader domain coverage despite being video-based.