[Paper Review] Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

Peter E. Clark, Isaac Cowhey+5 more|arXiv (Cornell University)|Mar 14, 2018

Topic Modeling23 references488 citations

TL;DR

Introduces ARC, a large, human-authored grade-school science QA dataset split into a challenging Challenge Set and an easier Easy Set, plus a 14M-sentence ARC Corpus and several neural baselines; results show current models struggle on the Challenge Set, highlighting the need for deeper reasoning.

ABSTRACT

We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering. Together, these constitute the AI2 Reasoning Challenge (ARC), which requires far more powerful knowledge and reasoning than previous challenges such as SQuAD or SNLI. The ARC question set is partitioned into a Challenge Set and an Easy Set, where the Challenge Set contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurence algorithm. The dataset contains only natural, grade-school science questions (authored for human tests), and is the largest public-domain set of this kind (7,787 questions). We test several baselines on the Challenge Set, including leading neural models from the SQuAD and SNLI tasks, and find that none are able to significantly outperform a random baseline, reflecting the difficult nature of this task. We are also releasing the ARC Corpus, a corpus of 14M science sentences relevant to the task, and implementations of the three neural baseline models tested. Can your model perform better? We pose ARC as a challenge to the community.

Motivation & Objective

• Motivate AI research in advanced question answering by emphasizing questions that require reasoning beyond surface cues.
• Provide a large, public dataset (ARC) with a clearly defined Challenge Set designed to defeat simple IR and co-occurrence baselines.
• Release a supporting science corpus (ARC Corpus) and baseline neural models to establish a starting point for the research community.

Proposed method

• Partition ARC into Challenge Set (hard) and Easy Set (easier) using retrieval- and co-occurrence-based baselines to define difficulty.
• Provide ARC Corpus of 14M science sentences to support knowledge-based reasoning.
• Adapt three neural QA models (DecompAttn, BiDAF, DGEM) for multiple-choice QA with retrieval-augmented inputs.
• Compare baselines including IR, PMI, and neural models on both Challenge and Easy sets to assess difficulty and knowledge requirements.
• Release code and leaderboard to enable community participation.

Experimental results

Research questions

• RQ1 Can standard IR/PMI baselines and leading neural QA models outperform random guessing on the ARC Challenge Set?
• RQ2 To what extent does the ARC Corpus assist retrieval-based baselines in answering Challenge questions?
• RQ3 Do neural models that perform well on SNLI/SQuAD significantly improve over random on the ARC Challenge Set?
• RQ4 Which knowledge and reasoning types are most critical for answering ARC Challenge questions?
• RQ5 How do performance patterns differ between the ARC Challenge Set and Easy Set?

Key findings

• No baseline model significantly outperforms random chance on the ARC Challenge Set (within tight confidence bounds).
• On the Easy Set, baselines generally achieve 55–65% accuracy, while Challenge Set performance remains near random, highlighting the difficulty.
• IR and PMI baselines perform poorly on the Challenge Set but can improve with the ARC Corpus for some questions, indicating knowledge is present but not easily exploitable by simple retrieval.
• Neural baselines (DecompAttn, BiDAF, DGEM) improve on the Easy Set but fail to surpass random on the Challenge Set, suggesting the need for more advanced retrieval and multi-hop reasoning strategies.
• ARC Corpus contains knowledge relevant to approximately 95% of Challenge questions, yet simple retrieval over this corpus is insufficient for the hardest questions.
• A notable gap exists in retrieval strategies that can combine multiple facts and perform multi-fact reasoning (chaining).