[Paper Review] WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challenge

Hongming Zhang, Xinran Zhao+1 more|arXiv (Cornell University)|May 12, 2020

Topic Modeling23 references4 citations

TL;DR

This paper introduces WinoWhy, a new diagnostic task that evaluates whether models truly understand commonsense reasoning or merely exploit statistical biases in the Winograd Schema Challenge (WSC). By collecting human-annotated reasons for correct answers and categorizing them into six knowledge types, the authors reveal that even state-of-the-art models achieve only 77.77% accuracy on WinoWhy, indicating limited comprehension of underlying commonsense knowledge despite strong performance on the original WSC task.

ABSTRACT

In this paper, we present the first comprehensive categorization of essential commonsense knowledge for answering the Winograd Schema Challenge (WSC). For each of the questions, we invite annotators to first provide reasons for making correct decisions and then categorize them into six major knowledge categories. By doing so, we better understand the limitation of existing methods (i.e., what kind of knowledge cannot be effectively represented or inferred with existing methods) and shed some light on the commonsense knowledge that we need to acquire in the future for better commonsense reasoning. Moreover, to investigate whether current WSC models can understand the commonsense or they simply solve the WSC questions based on the statistical bias of the dataset, we leverage the collected reasons to develop a new task called WinoWhy, which requires models to distinguish plausible reasons from very similar but wrong reasons for all WSC questions. Experimental results prove that even though pre-trained language representation models have achieved promising progress on the original WSC dataset, they are still struggling at WinoWhy. Further experiments show that even though supervised models can achieve better performance, the performance of these models can be sensitive to the dataset distribution. WinoWhy and all codes are available at: https://github.com/HKUST-KnowComp/WinoWhy.

Motivation & Objective

• To identify and categorize the essential types of commonsense knowledge required to solve Winograd Schema Challenge (WSC) questions.
• To investigate whether current models understand commonsense reasoning or simply exploit statistical biases in the WSC dataset.
• To develop a new diagnostic benchmark, WinoWhy, that evaluates models' ability to distinguish correct from highly similar but incorrect reasoning.
• To reveal the limitations of existing pre-trained and supervised models in comprehending the underlying rationale behind WSC decisions.

Proposed method

• Annotators provided detailed reasons for correct answers to each WSC question, explaining the commonsense knowledge used.
• Reasons were systematically categorized into six knowledge types: entity properties, temporal reasoning, spatial reasoning, causality, quantity, and social norms.
• A new task, WinoWhy, was created by pairing each WSC question with one correct reason and several plausible but incorrect reasons.
• Models were evaluated on their ability to classify the correct reason from a set of highly similar alternatives, framed as a text classification task.
• Multiple pre-trained models (e.g., BERT, RoBERTa, GPT-2, BERT-large) and an LSTM baseline were fine-tuned on the WinoWhy dataset using five-fold cross-validation.
• Two splitting strategies were used: by questions (Five-fold q) and by reasons (Five-fold r), to assess model robustness to data distribution shifts.

Experimental results

Research questions

• RQ1 What types of commonsense knowledge are essential for correctly solving Winograd Schema Challenge questions?
• RQ2 To what extent do current pre-trained language models understand the reasoning behind their predictions, rather than relying on dataset biases?
• RQ3 Can a model’s performance on WSC be reliably improved by fine-tuning on similar datasets, or is this due to overfitting to statistical patterns?
• RQ4 How robust are supervised models on WinoWhy when the training data distribution changes?

Key findings

• Even state-of-the-art models achieve only 77.77% accuracy on the WinoWhy task, indicating significant difficulty in distinguishing correct from plausible but incorrect reasoning.
• The best-performing supervised models show high standard deviations across cross-validation folds, especially under the 'Five-fold (r)' setting, suggesting sensitivity to data distribution.
• Fine-tuning on WSCR or WinoGrande improves performance on WSC but not on WinoWhy, implying that such gains may stem from dataset fitting rather than deeper commonsense understanding.
• The results suggest that current models are not truly understanding the reasoning behind WSC answers, but are instead relying on statistical cues in the training data.
• The WinoWhy benchmark reveals that existing models fail to generalize beyond surface-level patterns, highlighting the need for more robust reasoning mechanisms.