[Paper Review] Generating Followup Questions for Interpretable Multi-hop Question Answering

Christopher Malon, Bing Bai|arXiv (Cornell University)|Feb 27, 2020

Topic Modeling19 references4 citations

TL;DR

This paper proposes a neural framework for generating interpretable follow-up questions in multi-hop question answering, using a pointer-generator network to produce relevant subquestions based on partial evidence. The approach improves answer accuracy by 8.9% F1 over baselines in a two-hop setting, demonstrating that learned follow-up questions enhance retrieval specificity and model interpretability without requiring handcrafted rules.

ABSTRACT

We propose a framework for answering open domain multi-hop questions in which partial information is read and used to generate followup questions, to finally be answered by a pretrained single-hop answer extractor. This framework makes each hop interpretable, and makes the retrieval associated with later hops as flexible and specific as for the first hop. As a first instantiation of this framework, we train a pointer-generator network to predict followup questions based on the question and partial information. This provides a novel application of a neural question generation network, which is applied to give weak ground truth single-hop followup questions based on the final answers and their supporting facts. Learning to generate followup questions that select the relevant answer spans against downstream supporting facts, while avoiding distracting premises, poses an exciting semantic challenge for text generation. We present an evaluation using the two-hop bridge questions of HotpotQA.

Motivation & Objective

• To address the challenge of generating meaningful, interpretable follow-up questions in multi-hop QA, especially for bridge-type questions requiring intermediate reasoning.
• To develop a fully trained, end-to-end differentiable follow-up question generator that avoids reliance on handcrafted rules or heuristics.
• To evaluate the effectiveness of generated follow-up questions in improving answer extraction accuracy in a distractor setting.
• To establish a novel evaluation protocol for follow-up question generation using weak supervision from answer extractors.
• To explore how neural question generation can be adapted to produce questions that guide retrieval toward relevant supporting facts.

Proposed method

• A pointer-generator network is trained to generate follow-up questions based on the original question and partial supporting evidence from the first hop.
• The model uses a sequence-to-sequence architecture with coverage and copy mechanisms to generate fluent, factually grounded questions.
• Training is supervised using weak supervision: generated questions are evaluated based on whether a frozen single-hop answer extractor can correctly extract the final answer from the second-hop evidence.
• The system employs a controller module to classify retrieved passages as irrelevant, containing a final answer, or containing intermediate information.
• The pipeline iteratively reduces the original question into simpler subquestions, with answer extraction performed at each hop using a pre-trained single-hop model.
• An oracle setting is used to isolate the performance of the follow-up generator by assuming perfect retrieval of supporting facts.

Experimental results

Research questions

• RQ1 Can a neural question generator learn to produce follow-up questions that are both fluent and semantically aligned with the missing information needed to answer the original question?
• RQ2 Does generating follow-up questions improve answer accuracy compared to direct single-hop retrieval or trivial reusing of the original question?
• RQ3 How does the performance of the follow-up generator vary across different types of bridge questions in multi-hop QA?
• RQ4 To what extent can weak supervision from an answer extractor improve the quality of generated follow-up questions?
• RQ5 Can the generated follow-up questions effectively guide retrieval to avoid distracting premises while focusing on relevant supporting facts?

Key findings

• The trained follow-up question generator improved F1 score by 8.9 percentage points over the baseline that reuses the original question, demonstrating the value of context-aware subquestion generation.
• Using the strategy 'Q1 else Q2'—answering with Q1 if possible, otherwise using Q2—achieved the highest performance, with 34.7 F1 and 43.8 F1 on the filtered development set.
• The model successfully generated relevant follow-up questions in 70% of cases, as evidenced by correct answer extraction from second-hop evidence, even when the original question was ambiguous or insufficient.
• The system generated 1,180 follow-up queries in the second hop, with the answer extractor successfully extracting answers from 975 of them, showing high reliability in downstream use.
• The oracle evaluation setting confirmed that the follow-up generator significantly enhances retrieval specificity, as it avoids distracting premises while focusing on relevant facts.
• The approach outperformed a simple baseline that reused the original question, proving that learned follow-up questions are more effective than static queries in multi-hop reasoning.