[Paper Review] RACE: Large-scale ReAding Comprehension Dataset From Examinations

Guokun Lai, Qizhe Xie+3 more|arXiv (Cornell University)|Apr 15, 2017

Topic Modeling18 references112 citations

TL;DR

RACE introduces a large, human-created reading comprehension dataset from Chinese middle/high school English exams, emphasizing reasoning-heavy questions and broad topic coverage to challenge and evaluate machine comprehension systems. It provides baselines and human ceilings, showing a substantial gap between state-of-the-art models and human performance.

ABSTRACT

We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task. Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning. In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%). We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension. The dataset is freely available at http://www.cs.cmu.edu/~glai1/data/race/ and the code is available at https://github.com/qizhex/RACE_AR_baselines.

Motivation & Objective

• Motivate the need for a more challenging, diverse reading comprehension dataset beyond existing benchmarks.
• Create a large-scale, human-generated QA dataset grounded in English exams for ages 12–18 to evaluate reasoning ability.
• Ensure broad topic/style coverage and non-span-based answer options to demand deeper understanding.

Proposed method

• Collect English examination passages and questions from middle/high school levels (ages 12–18) in China.
• Filter and clean data to remove non-self-contained items, image-based answers, and format inconsistencies.
• Use human-generated questions and four-option multiple-choice answers where options may be non-spans from the passage.
• Split data into RACE-M (middle school) and RACE-H (high school) with train/dev/test sets.
• Evaluate baselines including Sliding Window, Stanford Attentive Reader, and Gated-Attention Reader, with SGD training and Glove embeddings.
• Provide human ceiling estimates via crowd worker labeling and manual validation of question validity.

Experimental results

Research questions

• RQ1 What is the distribution and difficulty of reasoning types required by RACE compared to existing QA datasets?
• RQ2 How do state-of-the-art machine comprehension models perform on this examination-derived dataset relative to human performance?
• RQ3 Does RACE’s broader domain/style coverage and non-span-based questions increase the need for multi-sentence and cross-sentence reasoning?
• RQ4 What are the characteristics and proportions of reasoning subdivisions (detail, whole-picture, summarization, attitude analysis, world knowledge) within RACE?

Key findings

• RACE contains 27,933 passages and 97,687 questions with a significant portion requiring reasoning beyond surface word matching.
• Human ceiling performance is high (~95%), while state-of-the-art models lag substantially (e.g., Stanford AR, GA around 43–44% on RACE), indicating a large room for improvement.
• Reasoning is the dominant challenge in RACE, with higher proportions of single- and multi-sentence reasoning than CNN/Daily Mail, SQUAD, and NEWSQA.
• Two novel reasoning subdivisions in RACE—passage summarization and attitude analysis—are underexplored in prior large-scale datasets.
• RACE demonstrates broad domain/style coverage (news, stories, ads, biographies, philosophy, etc.), making it a versatile benchmark for general reading comprehension.