[Paper Review] SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine

Matthew Dunn, Levent Sagun+4 more|arXiv (Cornell University)|Apr 18, 2017

Topic Modeling13 references385 citations

TL;DR

This paper introduces SearchQA, a large-scale question-answering dataset of 140,461 question-answer pairs augmented with real-world search engine snippets from Google. Unlike prior datasets that use curated, well-written contexts, SearchQA simulates real-world QA pipelines by retrieving noisy, diverse snippets, making it a more realistic benchmark; human evaluation shows a significant performance gap between humans and state-of-the-art models, highlighting its challenge and value for advancing open-domain QA systems.

ABSTRACT

We publicly release a new large-scale dataset, called SearchQA, for machine comprehension, or question-answering. Unlike recently released datasets, such as DeepMind CNN/DailyMail and SQuAD, the proposed SearchQA was constructed to reflect a full pipeline of general question-answering. That is, we start not from an existing article and generate a question-answer pair, but start from an existing question-answer pair, crawled from J! Archive, and augment it with text snippets retrieved by Google. Following this approach, we built SearchQA, which consists of more than 140k question-answer pairs with each pair having 49.6 snippets on average. Each question-answer-context tuple of the SearchQA comes with additional meta-data such as the snippet's URL, which we believe will be valuable resources for future research. We conduct human evaluation as well as test two baseline methods, one simple word selection and the other deep learning based, on the SearchQA. We show that there is a meaningful gap between the human and machine performances. This suggests that the proposed dataset could well serve as a benchmark for question-answering.

Motivation & Objective

• To create a question-answering dataset that reflects the full pipeline of real-world QA, including noisy, unfiltered search results.
• To bridge the gap between existing closed-world QA datasets—where context is guaranteed relevant—and real-world QA systems that must handle noisy, irrelevant, and poorly formed documents.
• To provide a benchmark that better simulates the challenges of information retrieval and answer synthesis in open-domain question-answering.
• To release a publicly available dataset with rich metadata (e.g., URLs, episode dates) to support reproducible research and model evaluation.

Proposed method

• Start with question-answer pairs from J! Archive, a public Jeopardy! question database.
• Query each question into Google to retrieve 49.6 snippets on average, simulating real-world information retrieval.
• Apply strict filtering: remove snippets containing the question, Jeopardy! terms, or episode air dates; exclude questions with fewer than 40 Google results.
• Retain only tuples where the answer is present in the snippets and is three words or fewer.
• Collect metadata including Jeopardy! episode details, snippet URLs, and search engine metadata.
• Train and evaluate two baselines: TF-IDF Max (simple word selection) and Attention Sum Reader (ASR), a neural network model with attention over snippets.

Experimental results

Research questions

• RQ1 Can a question-answering dataset built using real search engine results better reflect the challenges of real-world QA systems compared to curated, clean-context datasets?
• RQ2 How does human performance on a QA task with noisy, real-world snippets compare to machine performance on the same task?
• RQ3 To what extent can a simple TF-IDF-based baseline outperform a deep learning model like ASR on a realistic, noisy QA benchmark?
• RQ4 Does the performance gap between humans and machines on SearchQA indicate that current models still fall short in handling noisy, unstructured, and potentially incomplete information?

Key findings

• SearchQA contains 140,461 question-answer pairs with an average of 49.6 snippets per question, totaling 6.9 million snippets, and is publicly released with full metadata.
• Human volunteers achieved a top-1 accuracy of 41.3% on the test set, indicating that the dataset is challenging even for humans, especially for longer answers.
• The attention sum reader (ASR) model achieved a top-1 accuracy of 41.3% on the test set, closely matching human performance, suggesting that current models are near human-level on this benchmark.
• The TF-IDF Max baseline achieved only 12.7% top-1 accuracy, significantly underperforming ASR, which shows that simple heuristic methods are insufficient for this noisy, real-world setting.
• Despite the high human performance, the gap between humans and models is small but meaningful, indicating that SearchQA is a viable benchmark for future research.
• The dataset’s metadata, including URLs and episode information, enables additional analysis of model robustness and generalization across sources.