[Paper Review] DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications

Wei He, Kai Liu+11 more|arXiv (Cornell University)|Nov 14, 2017

Topic Modeling19 references51 citations

TL;DR

DuReader introduces a large-scale, real-world Chinese MRC dataset based on Baidu search logs and Zhidao, with diverse question types, and provides baseline results showing a gap to human performance.

ABSTRACT

This paper introduces DuReader, a new large-scale, open-domain Chinese ma- chine reading comprehension (MRC) dataset, designed to address real-world MRC. DuReader has three advantages over previous MRC datasets: (1) data sources: questions and documents are based on Baidu Search and Baidu Zhidao; answers are manually generated. (2) question types: it provides rich annotations for more question types, especially yes-no and opinion questions, that leaves more opportunity for the research community. (3) scale: it contains 200K questions, 420K answers and 1M documents; it is the largest Chinese MRC dataset so far. Experiments show that human performance is well above current state-of-the-art baseline systems, leaving plenty of room for the community to make improvements. To help the community make these improvements, both DuReader and baseline systems have been posted online. We also organize a shared competition to encourage the exploration of more models. Since the release of the task, there are significant improvements over the baselines.

Motivation & Objective

• Address real-world MRC challenges by using questions and documents sourced from Baidu Search and Baidu Zhidao.
• Provide rich question type annotations, including yes-no and opinion questions.
• Scale up to 200k questions, 1M documents, and 420k+ answers to reflect practical MRC scenarios.

Proposed method

• Collect questions from search logs via a classifier-assisted sampling pipeline.
• Assemble documents from Baidu Search and Baidu Zhidao and preserve full documents for paragraph-level reasoning.
• Annotate questions with six types (Entity, Description, YesNo) and two facets (Fact, Opinion).
• Crowdsource answers by summarizing and compiling supporting sentences across relevant documents.
• Evaluate using baseline MRC models (Match-LSTM, BiDAF) with a two-step paragraph selection and answer span procedures.
• Propose an opinion-aware evaluation for YesNo questions to align generated answers with opinion labels.

Experimental results

Research questions

• RQ1 How does MRC perform on real-world Chinese data drawn from search logs and a CQA platform?
• RQ2 Can models handle the six question types (Entity, Description, YesNo) across Fact/Opinion classifications?
• RQ3 What is the impact of using full documents versus single paragraphs on MRC performance?
• RQ4 How effective are current span-based MRC models on long, multi-document inputs?
• RQ5 What improvements arise when incorporating opinion-aware evaluation for YesNo questions?

Key findings

• DuReader contains 200k questions, 1M documents, and over 420k human-summarized answers, making it the largest Chinese MRC dataset at the time.
• Baseline models with paragraph selection (Match-LSTM, BiDAF) improve over the paragraph-baseline but still lag behind human performance.
• Gold-paragraph evaluation substantially boosts model performance, highlighting the importance of effective paragraph selection.
• YesNo and opinion questions are harder for models than description questions, and model performance varies across Baidu Search vs. Baidu Zhidao sources.
• An opinion-aware evaluation scheme improves assessment by pairing answers with explicit opinion labels (Yes/No/Depends).
• Human performance remains substantially higher than current models, indicating substantial room for methodological advances.