[Paper Review] Understanding and Improving Zero-shot Multi-hop Reasoning in Generative Question Answering

Zhengbao Jiang, Jun Araki+2 more|arXiv (Cornell University)|Oct 9, 2022

Topic Modeling4 citations

TL;DR

This paper investigates zero-shot multi-hop reasoning in generative question answering (QA) models, revealing that models fail to generalize from single-hop to multi-hop questions. It proposes two methods—concatenating single-hop questions and fine-tuning on SPARQL logical forms—to improve zero-shot multi-hop performance, achieving significant gains, especially when combined.

ABSTRACT

Generative question answering (QA) models generate answers to questions either solely based on the parameters of the model (the closed-book setting) or additionally retrieving relevant evidence (the open-book setting). Generative QA models can answer some relatively complex questions, but the mechanism through which they do so is still poorly understood. We perform several studies aimed at better understanding the multi-hop reasoning capabilities of generative QA models. First, we decompose multi-hop questions into multiple corresponding single-hop questions, and find marked inconsistency in QA models' answers on these pairs of ostensibly identical question chains. Second, we find that models lack zero-shot multi-hop reasoning ability: when trained only on single-hop questions, models generalize poorly to multi-hop questions. Finally, we demonstrate that it is possible to improve models' zero-shot multi-hop reasoning capacity through two methods that approximate real multi-hop natural language (NL) questions by training on either concatenation of single-hop questions or logical forms (SPARQL). In sum, these results demonstrate that multi-hop reasoning does not emerge naturally in generative QA models, but can be encouraged by advances in training or modeling techniques.

Motivation & Objective

• To investigate whether generative QA models can perform robust zero-shot multi-hop reasoning when trained only on single-hop questions.
• To analyze the consistency and correctness of model responses when decomposing multi-hop questions into single-hop components.
• To evaluate whether approximations of multi-hop questions—via concatenated single-hop questions or SPARQL queries—can enhance zero-shot multi-hop reasoning ability.
• To explore whether training on structured logical forms (SPARQL) enables generalization to natural language multi-hop questions.
• To identify limitations in current generative QA models' compositional reasoning and propose scalable training techniques to improve zero-shot multi-hop performance.

Proposed method

• Decompose multi-hop QA questions into chains of single-hop questions using heuristic methods to probe model consistency.
• Train generative QA models (UnifiedQA and RAG) on concatenated single-hop questions to approximate multi-hop reasoning patterns.
• Fine-tune models on SPARQL queries derived from multi-hop questions to teach explicit compositional reasoning structure.
• Combine both supervision signals (concatenated NL questions and SPARQL) to jointly improve zero-shot multi-hop reasoning.
• Use pre-trained language models with sequence-to-sequence generation to produce answers based on question embeddings and context (in open-book setting).
• Evaluate generalization by testing on both single-hop and multi-hop questions after fine-tuning, comparing performance to zero-shot baselines.

Experimental results

Research questions

• RQ1 Is the correctness of decomposed single-hop questions a necessary and sufficient condition for correctness of multi-hop questions?
• RQ2 Are answers to multi-hop questions and their component single-hop questions consistent across generative QA models?
• RQ3 Do models trained exclusively on single-hop questions demonstrate zero-shot generalization to multi-hop questions?
• RQ4 Can training on approximations of multi-hop questions—such as concatenated single-hop questions or SPARQL logical forms—improve zero-shot multi-hop reasoning capacity?
• RQ5 Does learning to execute SPARQL queries transfer to improved performance on natural language multi-hop questions?

Key findings

• Generative QA models exhibit marked inconsistency when answering decomposed single-hop questions versus their corresponding multi-hop counterparts, indicating flawed reasoning rather than reliable multi-hop inference.
• Models trained only on single-hop questions show poor zero-shot generalization to multi-hop questions, with performance dropping significantly on both closed-book (UnifiedQA) and open-book (RAG) settings.
• Training on concatenated single-hop questions improves zero-shot multi-hop performance, achieving 25.69 F1 on ComplexWebQuestions for UnifiedQA and 53.93 for RAG, showing that this method is an effective approximation of multi-hop reasoning.
• Fine-tuning on SPARQL queries enables models to generalize to natural language multi-hop questions, with a performance drop of only 7–15 F1 points compared to training on NL multi-hop questions, indicating strong transferability.
• Combining both supervision signals—concatenated NL questions and SPARQL—leads to further improvements, raising UnifiedQA’s F1 to 27.14 on multi-hop questions, demonstrating synergistic gains.
• Despite improvements, a large performance gap remains between zero-shot methods and models fine-tuned on actual multi-hop NL questions, highlighting the need for better approximations or modeling techniques.