[Paper Review] Exploring Unsupervised Pretraining and Sentence Structure Modelling for Winograd Schema Challenge

Yu-Ping Ruan, Xiaodan Zhu+4 more|arXiv (Cornell University)|Apr 22, 2019

Topic Modeling25 references18 citations

TL;DR

This paper achieves a new state-of-the-art accuracy of 71.1% on the Winograd Schema Challenge by jointly leveraging BERT-based unsupervised pretraining, fine-tuning on the Raham-Ng dataset, and incorporating syntactic dependency structures into the model. The approach demonstrates that dependency modeling consistently improves performance on harder non-associative cases, while fine-tuning is especially effective for simpler associative problems.

ABSTRACT

Winograd Schema Challenge (WSC) was proposed as an AI-hard problem in testing computers' intelligence on common sense representation and reasoning. This paper presents the new state-of-theart on WSC, achieving an accuracy of 71.1%. We demonstrate that the leading performance benefits from jointly modelling sentence structures, utilizing knowledge learned from cutting-edge pretraining models, and performing fine-tuning. We conduct detailed analyses, showing that fine-tuning is critical for achieving the performance, but it helps more on the simpler associative problems. Modelling sentence dependency structures, however, consistently helps on the harder non-associative subset of WSC. Analysis also shows that larger fine-tuning datasets yield better performances, suggesting the potential benefit of future work on annotating more Winograd schema sentences.

Motivation & Objective

• To improve performance on the Winograd Schema Challenge (WSC), an AI-hard benchmark for common sense reasoning.
• To investigate the effectiveness of state-of-the-art pretraining models like BERT in solving WSC without extensive rule-based or knowledge-engineered features.
• To analyze the impact of fine-tuning and syntactic structure modeling on different subsets of WSC (associative vs. non-associative).
• To evaluate how the size of the fine-tuning dataset influences model performance and to suggest future data collection efforts.

Proposed method

• Fine-tuned BERT-large on the Raham-Ng dataset, a human-annotated collection of Winograd schema examples.
• Integrated dependency parsing information into BERT through two methods: inside modeling (masking attention heads in specific layers) and outside modeling (using TransformerRNNs to encode dependency trees).
• Formulated WSC as a next-sentence prediction task to leverage BERT’s pretraining objectives.
• Evaluated performance across multiple splits: full WSC, associative/non-associative subsets, and unswitched/switched/consistent variants.
• Systematically varied the size of the fine-tuning data (from 0% to 100%) to assess data efficiency and scalability.
• Used masking strategies on BERT layers to explore which layers benefit most from dependency integration.

Experimental results

Research questions

• RQ1 To what extent does fine-tuning on a small annotated dataset improve BERT’s performance on the Winograd Schema Challenge?
• RQ2 How does incorporating syntactic dependency structures affect performance on the more challenging non-associative subset of WSC?
• RQ3 Does the performance gain from fine-tuning vary between associative and non-associative WSC instances?
• RQ4 How does the size of the fine-tuning dataset correlate with final model accuracy?
• RQ5 Which architectural integration method (inside vs. outside dependency modeling) yields better results in BERT for WSC?

Key findings

• The proposed model achieved a new state-of-the-art accuracy of 71.1% on the full Winograd Schema Challenge.
• Fine-tuning on the Raham-Ng dataset was critical for performance, with accuracy increasing from 52.7% (0% fine-tuning) to 71.1% (100% fine-tuning).
• Fine-tuning provided greater performance gains on the simpler associative subset (accuracy rose from 48.6% to 81.1%) compared to the non-associative subset (53.4% to 69.5%).
• Modeling dependency structures via masking the last five layers of BERT-large yielded the best performance (71.1%), demonstrating consistent improvement on the harder non-associative cases.
• The outside modeling approach (e.g., TransformerRNN) did not outperform the best inside modeling settings, especially for BERT-large.
• Larger fine-tuning datasets yielded better performance, with accuracy increasing monotonically from 66.7% (60%) to 71.1% (100%), suggesting that more annotated data could further improve results.