[Paper Review] The Defeat of the Winograd Schema Challenge

Vid Kocijan, Ernest Davis+3 more|arXiv (Cornell University)|Jan 7, 2022

Topic Modeling4 citations

TL;DR

This paper analyzes the decline of the Winograd Schema Challenge (WSC) as a benchmark for commonsense reasoning in NLP, demonstrating that state-of-the-art transformer-based models, especially RoBERTa and GPT-3, achieved over 90% accuracy on WSC273 by 2019. The authors argue that the challenge was effectively 'defeated' due to advances in pre-training, fine-tuning, and the emergence of large-scale surrogate datasets like WinoGrande, which enabled models to exploit statistical patterns rather than true commonsense understanding.

ABSTRACT

The Winograd Schema Challenge - a set of twin sentences involving pronoun reference disambiguation that seem to require the use of commonsense knowledge - was proposed by Hector Levesque in 2011. By 2019, a number of AI systems, based on large pre-trained transformer-based language models and fine-tuned on these kinds of problems, achieved better than 90% accuracy. In this paper, we review the history of the Winograd Schema Challenge and discuss the lasting contributions of the flurry of research that has taken place on the WSC in the last decade. We discuss the significance of various datasets developed for WSC, and the research community's deeper understanding of the role of surrogate tasks in assessing the intelligence of an AI system.

Motivation & Objective

• To analyze the historical trajectory and ultimate failure of the Winograd Schema Challenge as a benchmark for commonsense reasoning in AI.
• To investigate why large pre-trained language models achieved high performance on WSC despite its design to test deep understanding and commonsense knowledge.
• To examine the role of surrogate datasets like WinoGrande and WNLI in shifting research focus and enabling high accuracy without true reasoning.
• To evaluate the impact of fine-tuning, model scale, and data augmentation on performance across WSC benchmarks.
• To assess the implications of these results for the future of evaluating AI intelligence and the validity of surrogate benchmarks in NLP.

Proposed method

• Systematic review of the Winograd Schema Challenge's history, from its inception in 2011 to its high-accuracy results in 2019.
• Analysis of model architectures and training strategies used in top-performing systems, including BERT, RoBERTa, GPT-2, GPT-3, and T5.
• Evaluation of the role of fine-tuning on WSC-like data and external knowledge sources such as DPR, ConceptNet, and MaskedWiki.
• Examination of surrogate datasets like WinoGrande and WNLI, which enabled higher performance through data augmentation and multi-task learning.
• Comparison of results across datasets (WSC273, WNLI, WinoGrande, PDP) to track performance trends and model evolution.
• Identification of key trends: increasing model size, reliance on fine-tuning, and shift from small, high-quality datasets to large, synthetic ones.

Experimental results

Research questions

• RQ1 Why did the Winograd Schema Challenge fail to serve as a robust benchmark for commonsense reasoning despite its design to test deep understanding?
• RQ2 To what extent did pre-trained language models achieve high accuracy on WSC through statistical patterns rather than genuine commonsense reasoning?
• RQ3 How did the emergence of surrogate datasets like WinoGrande and WNLI contribute to the decline of the original WSC as a meaningful evaluation tool?
• RQ4 What role did fine-tuning and external knowledge sources play in improving model performance on WSC benchmarks?
• RQ5 What are the broader implications of this outcome for the evaluation of AI systems and the design of future benchmarks in NLP?

Key findings

• By 2019, RoBERTa-based models achieved 90.1% accuracy on the WSC273 benchmark, exceeding the 90% threshold that Levesque had set as a sign of human-level commonsense reasoning.
• The introduction of WinoGrande, a large-scale synthetic dataset, led to a shift in research focus, with most subsequent models only evaluating on this dataset rather than the original WSC.
• Fine-tuning on WSC-like data, especially with retrieval-augmented methods like DPR, significantly improved performance, with models like He et al. (2019) achieving 75.1% on WSC273 and 89% on WNLI.
• GPT-3 achieved 88.3% on WSC273 without task-specific fine-tuning, demonstrating that few-shot in-context learning could yield high performance on the challenge.
• The performance on WNLI, a smaller but widely used surrogate dataset, rose from 74.7% in 2019 to 85.6% by 2020, indicating rapid progress through data and model scaling.
• The trend toward larger models (e.g., GPT-3, T5-3B) and multi-task fine-tuning on diverse datasets (e.g., Khashabi et al., 2020) further accelerated performance gains, culminating in 91.3% accuracy on WinoGrande by 2021.