[Paper Review] Ask the Right Questions: Active Question Reformulation with Reinforcement Learning

Christian Buck, Jannis Bulian+5 more|arXiv (Cornell University)|May 22, 2017

Topic Modeling82 citations

TL;DR

This paper reframes Question Answering as a reinforcement learning problem, introducing AQA, an agent that reformulates questions to maximize QA quality through interaction with a black-box QA environment, and demonstrates substantial gains on SearchQA.

ABSTRACT

We frame Question Answering (QA) as a Reinforcement Learning task, an approach that we call Active Question Answering. We propose an agent that sits between the user and a black box QA system and learns to reformulate questions to elicit the best possible answers. The agent probes the system with, potentially many, natural language reformulations of an initial question and aggregates the returned evidence to yield the best answer. The reformulation system is trained end-to-end to maximize answer quality using policy gradient. We evaluate on SearchQA, a dataset of complex questions extracted from Jeopardy!. The agent outperforms a state-of-the-art base model, playing the role of the environment, and other benchmarks. We also analyze the language that the agent has learned while interacting with the question answering system. We find that successful question reformulations look quite different from natural language paraphrases. The agent is able to discover non-trivial reformulation strategies that resemble classic information retrieval techniques such as term re-weighting (tf-idf) and stemming.

Motivation & Objective

• Motivate improving QA performance by iterative question reformulation in a black-box setting.
• Develop an end-to-end RL-driven reformulation agent that operates between users and a QA environment.
• Demonstrate that learned reformulations can outperform state-of-the-art QA models and heuristic baselines.
• Analyze the linguistic nature of agent reformulations and relate them to classic IR techniques.

Proposed method

• Formulate QA as an RL problem where the reformulation model generates multiple rewrites of an initial question.
• Use a sequence-to-sequence reformulator pretrained on multilingual translation and refined with RL using policy gradient and a baseline for variance reduction.
• Employ a BiDAF-based environment as a black-box QA system to return answers and rewards based on answer quality (token-level F1).
• Train an answer selector (CNN-based) to pick the best answer from multiple reformulations using F1-derived signals.
• Pretrain reformulator via multilingual MT data (zero-shot translation) and refine with monolingual paraphrase data (Paralex) to improve rewrite quality.
• Evaluate with a CNN-based answer selector and compare against MI-SubQuery, Base-NMT, and human baselines on SearchQA.

Experimental results

Research questions

• RQ1 Can an RL-powered reformulation agent improve QA performance by querying a black-box QA system with multiple natural-language rewrites?
• RQ2 What kind of reformulations does the agent learn, and do they resemble classical IR techniques (e.g., tf-idf weighting, stemming)?
• RQ3 Does an end-to-end trained AQA system outperform baselines and human performance on the SearchQA dataset?
• RQ4 How does the language produced by the agent differ from natural paraphrases, and how does it relate to QA modality?
• RQ5 What impact does pretraining strategy (multilingual MT → monolingual paraphrase data) have on final QA performance?

Key findings

• AQA improves BiDAF by 11.4 percentage points in absolute F1 on the SearchQA test set (32% relative F1 improvement).
• Using the top hypothesis from the reformulator yields an immediate 2.2 F1 improvement over BiDAF with original questions.
• A dedicated CNN answer selector trained on reformulation candidates outperforms both MI-SubQuery and Base-NMT baselines by about 3% in F1 across conditions.
• Oracle-style evaluation shows that the potential headroom from reformulations is substantial (oracle EM near 50, oracle F1 near 58).
• The agent learns non-trivial reformulation strategies that resemble IR techniques like term re-weighting and stemming, diverging from natural paraphrasing.
• Reformulations can be longer and more term-rich (higher DF terms, increased duplication) to aid alignment in neural QA encoders.