[Paper Review] A Simple Method for Commonsense Reasoning

Trieu H. Trinh, Quoc V. Le|arXiv (Cornell University)|Jun 7, 2018

Natural Language Processing Techniques38 references312 citations

TL;DR

The authors show that unsupervised large language models trained on diverse unlabeled corpora can solve Winograd Schema challenges and Pronoun Disambiguation problems by scoring candidate substitutions, achieving state-of-the-art accuracy without hand-engineered features or annotated knowledge bases.

ABSTRACT

Commonsense reasoning is a long-standing challenge for deep learning. For example, it is difficult to use neural networks to tackle the Winograd Schema dataset (Levesque et al., 2011). In this paper, we present a simple method for commonsense reasoning with neural networks, using unsupervised learning. Key to our method is the use of language models, trained on a massive amount of unlabled data, to score multiple choice questions posed by commonsense reasoning tests. On both Pronoun Disambiguation and Winograd Schema challenges, our models outperform previous state-of-the-art methods by a large margin, without using expensive annotated knowledge bases or hand-engineered features. We train an array of large RNN language models that operate at word or character level on LM-1-Billion, CommonCrawl, SQuAD, Gutenberg Books, and a customized corpus for this task and show that diversity of training data plays an important role in test performance. Further analysis also shows that our system successfully discovers important features of the context that decide the correct answer, indicating a good grasp of commonsense knowledge.

Motivation & Objective

• Motivate commonsense reasoning as a low-supervision problem where labeled data is scarce.
• Propose a simple method using language models to score candidate substitutions in Winograd Schema and PDP tasks.
• Demonstrate that ensemble models trained on diverse corpora outperform prior state-of-the-art approaches.
• Analyze how scoring strategy and training data diversity affect performance on reasoning benchmarks.

Proposed method

• Substitute the pronoun with each candidate reference in a sentence and score the resulting sentences using language models.
• Compare full sentence probability (Score_full) with conditional probability of the tail given the substitution (Score_partial).
• Train word- and character-based LMs on large unlabeled corpora (LM-1-Billion, CommonCrawl, SQuAD, Gutenberg, STORIES) and ensemble their outputs.
• Evaluate on PDP-60 and WSC-273 to measure reasoning capabilities without annotated knowledge bases.
• Explore a customized STORIES corpus derived from CommonCrawl to further boost performance for Winograd Schema tasks.
• Analyze keyword-like features by inspecting per-token probability ratios to identify decision-critical words.

Experimental results

Research questions

• RQ1 Can unsupervised language models learn enough commonsense reasoning from large unlabeled corpora to solve Winograd Schema and pronoun disambiguation tasks?
• RQ2 Does scoring method (full vs partial) affect reasoning performance, and how does training data diversity influence results?
• RQ3 What is the impact of corpus choice on LM performance for commonsense tasks, and can a Story-like corpus provide additional gains?
• RQ4 Do ensembles of multiple LMs trained on diverse corpora outperform single models or models using knowledge bases?
• RQ5 Can the model identify keywords or special words that drive the Winograd Schema decisions?

Key findings

• Single-model LMs outperform previous methods on PDP-60 when using full scoring, with Word-LM-full achieving 60.0% accuracy.
• Ensembles of unsupervised LMs surpass the prior best PDP-60 result (66.7%) with 70.0% accuracy.
• On WSC-273, Word-LM-full reaches 53.8% accuracy and Word-LM-partial reaches 56.4%.
• Ensembling 10 LMs trained on diverse corpora yields 61.5% accuracy on WSC-273, and 63.7% with an expanded ensemble including STORIES.
• Partial scoring consistently outperforms full scoring on WSC-273 and PDP datasets, and normalization of Score_full helps on PDP-122.
• Training on the STORIES corpus yields strong single-model performance (62.6%), and adding STORY-based models to the ensemble improves final WSC-273 accuracy to 63.7%.
• Diversity of training data is beneficial; ensembles trained on a diverse set of corpora outperform those trained on a single corpus.