[Paper Review] Reading Wikipedia to Answer Open-Domain Questions

Danqi Chen, Adam Fisch+2 more|arXiv (Cornell University)|Mar 31, 2017

Topic Modeling31 references90 citations

TL;DR

DrQA combines a fast document retriever over Wikipedia with a neural document reader to perform open-domain question answering using Wikipedia as the sole knowledge source, achieving strong results on multiple QA benchmarks and demonstrating benefits from multitask and distant supervision training.

ABSTRACT

This paper proposes to tackle open- domain question answering using Wikipedia as the unique knowledge source: the answer to any factoid question is a text span in a Wikipedia article. This task of machine reading at scale combines the challenges of document retrieval (finding the relevant articles) with that of machine comprehension of text (identifying the answer spans from those articles). Our approach combines a search component based on bigram hashing and TF-IDF matching with a multi-layer recurrent neural network model trained to detect answers in Wikipedia paragraphs. Our experiments on multiple existing QA datasets indicate that (1) both modules are highly competitive with respect to existing counterparts and (2) multitask learning using distant supervision on their combination is an effective complete system on this challenging task.

Motivation & Objective

• Motivate open-domain QA where Wikipedia is the sole knowledge source and identify the challenges of retrieving relevant articles and reading them for span-based answers.
• Develop a two-component system (Document Retriever and Document Reader) to tackle retrieval and machine comprehension at scale.
• Evaluate the system on multiple QA datasets to assess transfer and the utility of distant supervision and multitask learning.

Proposed method

• Document Retriever: an efficient retrieval module using bigram hashing and TF-IDF scoring to return a small set of relevant Wikipedia articles.
• Document Reader: a multi-layer bidirectional LSTM neural network that encodes paragraphs and questions, uses features including word embeddings, exact-match signals, POS/NER/TF, and an aligned question embedding, and predicts span starts/ends for answers.
• Training: use SQuAD for primary training; generate distantly supervised data by associating paragraphs with non-SQuAD QA pairs; explore multitask learning by combining DS data with SQuAD.
• Prediction: select the best answer span across retrieved documents by scoring start and end positions with bilinear terms and choosing a span within a paragraph (up to 15 tokens).
• Evaluation: assess components separately on SQuAD, then evaluate the full DrQA system on open-domain QA tasks across SQuAD development set, CuratedTREC, WebQuestions, and WikiMovies.

Experimental results

Research questions

• RQ1 Can Wikipedia alone support open-domain factoid QA with a modular pipeline that retrieves and reads texts to extract answer spans?
• RQ2 How do retrieval quality and paragraph-level machine comprehension interact to affect overall QA performance on diverse datasets?
• RQ3 Does multitask training with distant supervision improve open-domain QA performance beyond single-dataset training?
• RQ4 What is the gap between reading a single correct paragraph (machine reading) and reading multiple paragraphs/documents from Wikipedia in open-domain QA?

Key findings

• Document Retriever with bigram hashing outperforms the built-in Wikipedia search API in locating answer-containing articles.
• Document Reader achieves strong SQuAD results as a single model, reaching 70.0% Exact Match and 79.0% F1 on the test set, surpassing several published systems.
• DrQA’s full open-domain QA system yields competitive results across SQuAD, CuratedTREC, WebQuestions, and WikiMovies when trained with multitask distant supervision, outperforming single-task training in many cases.
• Distant supervision and multitask training contribute meaningful gains beyond training only on SQuAD, indicating benefits from additional domain-relevant data in the same knowledge source.
• When predicting open-domain answers, the system experiences performance drops relative to paragraph-reading since questions in SQuAD are often tailored to a specific paragraph, highlighting the challenge of end-to-end open-domain QA.