[Paper Review] BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

Christopher Clark, Kenton Lee+4 more|arXiv (Cornell University)|May 24, 2019

Topic Modeling39 references209 citations

TL;DR

The paper introduces BoolQ, a dataset of naturally occurring yes/no questions paired with Wikipedia passages, showing they require substantial inference and transfer learning improves performance over baselines.

ABSTRACT

In this paper we study yes/no questions that are naturally occurring --- meaning that they are generated in unprompted and unconstrained settings. We build a reading comprehension dataset, BoolQ, of such questions, and show that they are unexpectedly challenging. They often query for complex, non-factoid information, and require difficult entailment-like inference to solve. We also explore the effectiveness of a range of transfer learning baselines. We find that transferring from entailment data is more effective than transferring from paraphrase or extractive QA data, and that it, surprisingly, continues to be very beneficial even when starting from massive pre-trained language models such as BERT. Our best method trains BERT on MultiNLI and then re-trains it on our train set. It achieves 80.4% accuracy compared to 90% accuracy of human annotators (and 62% majority-baseline), leaving a significant gap for future work.

Motivation & Objective

• Motivate and construct a dataset of naturally occurring yes/no questions to test inferential reading comprehension.
• Characterize the kinds of reasoning necessary to answer these questions.
• Evaluate transfer learning strategies for yes/no QA and identify effective pre-training sources.

Proposed method

• Assemble BoolQ by extending Natural Questions data collection to yes/no questions with passages indicating the answer.
• Annotate questions for answerability and yes/no labels with passage excerpts.
• Experiment with transfer learning from entailment, paraphrase, and extractive QA sources, including unsupervised pre-training.
• Fine-tune models (Recurrent, BERT, GPT, etc.) on BoolQ and compare single-step vs multi-step pre-training.
• Analyze question/passage signal, and study how pre-training on datasets like MultiNLI affects BoolQ performance.

Experimental results

Research questions

• RQ1 Can naturally occurring yes/no questions be answered reliably from short passages requiring non-trivial inference?
• RQ2 Which external data sources (entailment, QA, paraphrase, unsupervised pre-training) best transfer to BoolQ?
• RQ3 How does transfer learning compare to training on BoolQ alone, and how does multi-step pre-training impact performance?
• RQ4 What inference types are most critical for BoolQ (paraphrase, factual reasoning, implicit/missing-mention, etc.)?
• RQ5 What is the gap between machine performance and human accuracy on BoolQ?

Key findings

• BoolQ is challenging: best model achieves 80.43% accuracy vs. 90% human accuracy and 62.31% majority baseline.
• Transferring from entailment data (MultiNLI, SNLI) generally yields stronger gains than from paraphrase or extractive QA data.
• Unsupervised pre-training with BERT and a two-step regime (pre-train on MultiNLI, then fine-tune on BoolQ) yields substantial gains, with MultiNLI contributing orthogonal signal to BERT.
• A single-step transfer from MultiNLI outperforms many other sources, and the combination of MultiNLI pre-training followed by BoolQ fine-tuning achieves the top results.
• Question-only models underperform, indicating passages carry essential information for answer prediction.
• Transfer from tasks that resemble BoolQ’s format (e.g., sentence pairs) is not always superior to entailment-based transfer; entailment data remains highly beneficial.