[Paper Review] WinoGrande: An Adversarial Winograd Schema Challenge at Scale

Keisuke Sakaguchi, Ronan Le Bras+2 more|arXiv (Cornell University)|Jul 24, 2019

Topic ModelingComputer Science45 references84 citations

TL;DR

WinoGrande scales up Winograd-like pronoun resolution problems to 44k items and uses AfLite to debias data, showing substantial gaps between human and model performance and enabling transfer learning to related benchmarks.

ABSTRACT

The Winograd Schema Challenge (WSC) (Levesque, Davis, and Morgenstern 2011), a benchmark for commonsense reasoning, is a set of 273 expert-crafted pronoun resolution problems originally designed to be unsolvable for statistical models that rely on selectional preferences or word associations. However, recent advances in neural language models have already reached around 90% accuracy on variants of WSC. This raises an important question whether these models have truly acquired robust commonsense capabilities or whether they rely on spurious biases in the datasets that lead to an overestimation of the true capabilities of machine commonsense. To investigate this question, we introduce WinoGrande, a large-scale dataset of 44k problems, inspired by the original WSC design, but adjusted to improve both the scale and the hardness of the dataset. The key steps of the dataset construction consist of (1) a carefully designed crowdsourcing procedure, followed by (2) systematic bias reduction using a novel AfLite algorithm that generalizes human-detectable word associations to machine-detectable embedding associations. The best state-of-the-art methods on WinoGrande achieve 59.4-79.1%, which are 15-35% below human performance of 94.0%, depending on the amount of the training data allowed. Furthermore, we establish new state-of-the-art results on five related benchmarks - WSC (90.1%), DPR (93.1%), COPA (90.6%), KnowRef (85.6%), and Winogender (97.1%). These results have dual implications: on one hand, they demonstrate the effectiveness of WinoGrande when used as a resource for transfer learning. On the other hand, they raise a concern that we are likely to be overestimating the true capabilities of machine commonsense across all these benchmarks. We emphasize the importance of algorithmic bias reduction in existing and future benchmarks to mitigate such overestimation.

Motivation & Objective

• Investigate whether large-scale models truly possess commonsense reasoning or rely on dataset biases.
• Create a large, harder WSC-inspired dataset to challenge current models.
• Develop and apply a bias-reduction method (AfLite) to mitigate dataset-specific artifacts.
• Evaluate transfer learning capabilities from WinoGrande to other commonsense benchmarks.

Proposed method

• Crowdsourced generation of twin-sentence pronoun-disambiguation problems guided by topic anchors to increase diversity.
• AfLite: a lightweight adversarial filtering algorithm using RoBERTa embeddings and an ensemble of linear classifiers to remove bias-heavy instances.
• Comparison of debiased vs. all data setups to assess bias effects using KL divergence and PCA visualizations.
• Baseline and state-of-the-art models (WKH, Ensemble LMs, BERT, RoBERTa, with/without DPR fine-tuning) on WinoGrande debiased and all.
• Transfer-learning experiments fine-tuning RoBERTa on WinoGrande to evaluate gains on WSC, PDP, COPA, KnowRef, and Winogender.

Experimental results

Research questions

• RQ1 Can crowdsourced WSC-inspired problems be scaled to tens of thousands while preserving difficulty for AI and ensuring human solvability?
• RQ2 Do dataset-specific biases inflate model performance on WSC-style tasks, and can AfLite mitigate these biases?
• RQ3 How does WinoGrande debiasing affect model performance and transfer learning to related benchmarks?
• RQ4 To what extent can a model trained on WinoGrande transfer to other commonsense reasoning datasets?],
• RQ5 key_findings・1-experimentation_quantitative_summaryAdded note
• RQ6 key_findings":["Best RoBERTa performance on debiased WinoGrande test set is 79.1% (dev 79.3%).","Human performance on debiased WinoGrande exceeds 94.0% accuracy, far above model scores.","AfLite debiasing dramatically reduces KL divergence between label distributions, indicating reduced dataset-specific bias.","RoBERTa fine-tuned on WinoGrande improves state-of-the-art results on WSC, DPR, COPA, KnowRef, and Winogender.","WinoGrande enables transfer learning; RoBERTa-WinoGrande achieves 90.1% on WSC-related tasks, 93.1% on DPR, 90.6% on COPA, 85.6% on KnowRef, and 97.1% on Winogender (relative to respective baselines).","Results suggest substantial biases in existing benchmarks and the need for algorithmic bias reduction to better gauge true commonsense capabilities."],
• RQ7 table_headers
• RQ8 table_rows
• RQ9 Main results table (Dev/Test accuracy on WinoGrande-debiased)
• RQ10 ["RoBERTa","79.3","79.1"],["BERT","65.8","64.9"],["Ensemble LMs","53.0","50.9"],["WKH","49.4","49.6"],["RoBERTa (local context)","52.1","50.0"],["BERT (local context)","52.5","51.9"],["BERT-DPR Star","50.2","51.0"],["RoBERTa-DPR Star","59.4","58.9"],["Human Perf.","94.1","94.0"]