[Paper Review] R$^3$: Reinforced Reader-Ranker for Open-Domain Question Answering

Shuohang Wang, Mo Yu+8 more|arXiv (Cornell University)|Aug 31, 2017

Topic Modeling34 references87 citations

TL;DR

R3 introduces a Reinforced Ranker-Reader system for open-domain QA that jointly trains a passage-ranker and a reader via reinforcement learning to maximize end-question-answer performance, achieving state-of-the-art on several datasets.

ABSTRACT

In recent years researchers have achieved considerable success applying neural network methods to question answering (QA). These approaches have achieved state of the art results in simplified closed-domain settings such as the SQuAD (Rajpurkar et al., 2016) dataset, which provides a pre-selected passage, from which the answer to a given question may be extracted. More recently, researchers have begun to tackle open-domain QA, in which the model is given a question and access to a large corpus (e.g., wikipedia) instead of a pre-selected passage (Chen et al., 2017a). This setting is more complex as it requires large-scale search for relevant passages by an information retrieval component, combined with a reading comprehension model that "reads" the passages to generate an answer to the question. Performance in this setting lags considerably behind closed-domain performance. In this paper, we present a novel open-domain QA system called Reinforced Ranker-Reader $(R^3)$, based on two algorithmic innovations. First, we propose a new pipeline for open-domain QA with a Ranker component, which learns to rank retrieved passages in terms of likelihood of generating the ground-truth answer to a given question. Second, we propose a novel method that jointly trains the Ranker along with an answer-generation Reader model, based on reinforcement learning. We report extensive experimental results showing that our method significantly improves on the state of the art for multiple open-domain QA datasets.

Motivation & Objective

• Motivation to improve open-domain QA beyond closed passages by effectively ranking relevant passages.
• Propose a two-component framework (Ranker and Reader) that separates passage selection from answer extraction.
• Enable end-to-end optimization of passage ranking with respect to final answer quality.
• Demonstrate strong empirical gains across multiple open-domain QA datasets.

Proposed method

• Two-component architecture: a Ranker that selects the most answer-entailing passage and a Reader that extracts the answer from that passage.
• Both components use Match-LSTM based representations to compare questions against passages via attention mechanisms.
• Ranker is trained with REINFORCE using a reward based on how well the Reader’s extracted answer matches the ground-truth.
• Reader is trained with SGD/backprop to maximize the likelihood of the correct answer span within the selected passage.
• Training jointly combines reinforcement learning for ranking with supervised optimization for reading, and uses negative sampling to stabilize Reader training.

Experimental results

Research questions

• RQ1 Can a separate Ranker trained with reinforcement learning improve open-domain QA by selecting more answer-worthy passages?
• RQ2 Does joint training of Ranker and Reader outperform single-reader or non-reinforcement baselines in open-domain QA?
• RQ3 How close can the Ranker-Reader approach get to an oracle ranking of passages containing the correct answer?

Key findings

• R3 achieves state-of-the-art results on multiple open-domain QA datasets.
• Joint RL-based Ranker and supervised Reader outperform internal baselines (Single Reader and Simple Ranker-Reader) and several public baselines.
• The Ranker trained with RL improves top-1/top-3 passage recall compared with non-RL rankers, aiding answer extraction.
• Using a bounded reward with F1-based guidance reduces gradient variance and stabilizes training.
• The model benefits from pre-training with a simpler Ranker-Reader variant and outperforms when jointly trained.
• On several datasets, R3 significantly improves over baselines, demonstrating the value of end-to-end optimization for passage ranking in open-domain QA.