[Paper Review] ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning

Weihao Yu, Zihang Jiang+2 more|arXiv (Cornell University)|Feb 11, 2020

Topic Modeling40 references127 citations

TL;DR

ReClor introduces a large multiple-choice reading comprehension dataset focused on logical reasoning, splits test data into biased EASY and non-biased HARD sets, and shows that state-of-the-art models excel on EASY but struggle on HARD, with transfer learning (RACE pre-training) offering notable gains but still below human performance on HARD.

ABSTRACT

Recent powerful pre-trained language models have achieved remarkable performance on most of the popular datasets for reading comprehension. It is time to introduce more challenging datasets to push the development of this field towards more comprehensive reasoning of text. In this paper, we introduce a new Reading Comprehension dataset requiring logical reasoning (ReClor) extracted from standardized graduate admission examinations. As earlier studies suggest, human-annotated datasets usually contain biases, which are often exploited by models to achieve high accuracy without truly understanding the text. In order to comprehensively evaluate the logical reasoning ability of models on ReClor, we propose to identify biased data points and separate them into EASY set while the rest as HARD set. Empirical results show that state-of-the-art models have an outstanding ability to capture biases contained in the dataset with high accuracy on EASY set. However, they struggle on HARD set with poor performance near that of random guess, indicating more research is needed to essentially enhance the logical reasoning ability of current models.

Motivation & Objective

• Introduce ReClor, a reading comprehension dataset that requires logical reasoning sourced from GMAT/LSAT exams.
• Identify data biases by separating biased (EASY) from non-biased (HARD) data to comprehensively evaluate reasoning ability.
• Evaluate state-of-the-art pre-trained models on ReClor to assess reliance on dataset biases versus true reasoning.
• Examine transfer learning by fine-tuning on RACE before training on ReClor to assess gains in reasoning tasks.

Proposed method

• Assemble 6,138 logical reasoning questions from GMAT/LSAT sources and transform to four-option MC format by shuffling options.
• Identify biased data by inputting only answer options to strong baselines with multiple seeds and intersecting predictions to define EASY set.
• Compare model performance on EASY vs HARD subsets to gauge reasoning vs bias exploitation.
• Train and evaluate baseline models (GPT, GPT-2, BERT, XLNet, RoBERTa) under various input settings (C/Q/A) and with/without RACE pre-training.
• Analyze question-type distributions (17 types) and examine performance across HARD vs EASY to understand reasoning challenges.
• Demonstrate transfer learning by fine-tuning on RACE before ReClor and report improvements, especially on HARD.

Experimental results

Research questions

• RQ1 Can current NLP models perform genuine logical reasoning on a dataset constructed from standardized exams?
• RQ2 Do state-of-the-art models primarily exploit dataset biases, and can EASY/HARD splits reveal this behavior?
• RQ3 Does pre-training or fine-tuning on a related exam-style dataset (RACE) improve reasoning performance on ReClor, particularly on HARD cases?

Key findings

• ReClor is of comparable size to similar MRC datasets, with a large vocabulary and relatively concise context passages.
• Models perform well on the EASY set (bias-driven questions) but struggle on the HARD set, approaching random chance on HARD.
• Transformer-based models achieve higher accuracy on EASY (~60+%), but reach around 30-40% on HARD, indicating limited real logical reasoning.
• Fine-tuning on RACE before ReClor yields substantial gains, especially on HARD, though still below graduate-student/human performance.
• Humans maintain more consistent performance across EASY and HARD, highlighting a gap between machine bias exploitation and true reasoning.
• Analysis shows bias signals include lexical cues and option length differences, and certain question types (e.g., strengthen/weaken, role) are particularly challenging for HARD.