[Paper Review] Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning

Lifu Huang, Ronan Le Bras+2 more|arXiv (Cornell University)|Aug 31, 2019

Topic Modeling89 citations

TL;DR

Cosmos QA introduces a large-scale multiple-choice dataset that requires contextual commonsense reasoning beyond explicit text, and evaluates various baselines including a new BERT-based multiway attention model, revealing a substantial gap between machine and human performance.

ABSTRACT

Understanding narratives requires reading between the lines, which in turn, requires interpreting the likely causes and effects of events, even when they are not mentioned explicitly. In this paper, we introduce Cosmos QA, a large-scale dataset of 35,600 problems that require commonsense-based reading comprehension, formulated as multiple-choice questions. In stark contrast to most existing reading comprehension datasets where the questions focus on factual and literal understanding of the context paragraph, our dataset focuses on reading between the lines over a diverse collection of people's everyday narratives, asking such questions as "what might be the possible reason of ...?", or "what would have happened if ..." that require reasoning beyond the exact text spans in the context. To establish baseline performances on Cosmos QA, we experiment with several state-of-the-art neural architectures for reading comprehension, and also propose a new architecture that improves over the competitive baselines. Experimental results demonstrate a significant gap between machine (68.4%) and human performance (94%), pointing to avenues for future research on commonsense machine comprehension. Dataset, code and leaderboard is publicly available at https://wilburone.github.io/cosmos.

Motivation & Objective

• Motivate research on reading comprehension that requires contextual commonsense reasoning beyond explicit text spans.
• Provide a large-scale benchmark (Cosmos QA) with diverse everyday narratives to study cause-effect, counterfactuals, and social commonsense.
• Evaluate state-of-the-art MRC models and propose architectures tailored to contextual commonsense inference.

Proposed method

• Construct Cosmos QA with 35,588 question sets from blog narratives and validated human-generated questions and answers.
• Formulate the task as multiple-choice, with None-of-the-above for unanswerable cases.
• Fine-tune pre-trained models (notably BERT) on Cosmos QA and compare baselines including traditional MRC models and GPT-based approaches.
• Propose a BERT variant with multiway attention that models interactions among paragraph, question, and candidate answers.
• Define a training objective that uses a softmax over four candidate answers with a cross-entropy loss.
• Conduct knowledge-transfer experiments by fine-tuning on related datasets (RACE, SWAG) and then evaluating on Cosmos.

Experimental results

Research questions

• RQ1 How well do existing reading comprehension models perform on questions requiring contextual commonsense reasoning?
• RQ2 Does incorporating multiway attention over paragraph, question, and answer improve performance on Cosmos QA?
• RQ3 What is the impact of fine-tuning with related datasets (RACE, SWAG) on Cosmos QA performance?
• RQ4 How often are correct Cosmos QA answers not explicitly stated in the context paragraph, and what types of commonsense reasoning are most challenging?

Key findings

• Cosmos QA achieves 68.4% test accuracy with the best model, vs. 94.0% human performance.
• Most correct answers (over 83%) are not explicitly stated in the paragraph, highlighting the need for contextual commonsense inference.
• A BERT-based multiway attention model improves over standard BERT fine-tuning (67.1% to 68.4% on test).
• Pre-trained language models with fine-tuning outperform traditional reading comprehension models by a sizable margin.
• Knowledge transfer: direct transfer from RACE helps more than SWAG; sequential fine-tuning on SWAG before Cosmos yields strong gains.
• Ablation shows removing interaction across P, Q, A hurts performance; questions contribute less when ablated, indicating robust context dependence.