[Paper Review] HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

Zhilin Yang, Peng Qi+5 more|arXiv (Cornell University)|Sep 25, 2018

Topic Modeling16 references111 citations

TL;DR

HotpotQA introduces 113k Wikipedia-based QA pairs requiring multi-hop reasoning without KB constraints, provides sentence-level supporting facts for explainability, and adds novel comparison questions.

ABSTRACT

Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. We introduce HotpotQA, a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) we provide sentence-level supporting facts required for reasoning, allowing QA systems to reason with strong supervision and explain the predictions; (4) we offer a new type of factoid comparison questions to test QA systems' ability to extract relevant facts and perform necessary comparison. We show that HotpotQA is challenging for the latest QA systems, and the supporting facts enable models to improve performance and make explainable predictions.

Motivation & Objective

• Motivate the need for datasets that require multi-hop reasoning across multiple documents in natural language.
• Provide a large-scale, Wikipedia-based QA dataset without fixed KB schemas to diversify questions and answers.
• Enable strong supervision by collecting sentence-level supporting facts to explain predictions.
• Introduce comparison questions to test arithmetic and cross-entity reasoning.
• Offer benchmarks that assess both QA accuracy and explainability through supporting facts.

Proposed method

• Construct a Wikipedia hyperlink graph from first-paragraph hyperlinks to identify bridge entities for multi-hop reasoning.
• Generate candidate paragraph pairs via edges in the graph and curated bridge entities to create meaningful multi-hop questions.
• Collect supporting facts sentences that justify answers for strong supervision and explanation.
• Create a new question type: comparison questions to test factual and numerical reasoning across entities.
• Develop an end-to-end QA model with a multi-task objective that predicts both the answer span and supporting facts, including a yes/no/span decision branch for yes/no questions.
• Evaluate QA performance under distractor and full-wiki retrieval settings with both answer accuracy and explainability metrics.

Experimental results

Research questions

• RQ1 How well do existing QA models perform on truly multi-hop, text-based questions that require reasoning across multiple documents?
• RQ2 Can providing sentence-level supporting facts improve both QA accuracy and the explainability of model predictions?
• RQ3 What is the impact of retrieval difficulty (distractor vs full Wiki) on multi-hop QA performance?
• RQ4 How do comparison questions and yes/no questions affect QA challenges and model capabilities?
• RQ5 What is the distribution and nature of multi-hop reasoning types required by HotpotQA?

Key findings

• A large-scale dataset (112,779 examples) requiring multi-hop reasoning over multiple documents.
• Two benchmark settings (distractor and full wiki) to test retrieval and reasoning under different noise levels.
• Strong supervision over supporting facts improves QA performance and enables explainability (supporting-fact prediction).
• Baseline models are significantly below human performance, especially in full-wiki retrieval, highlighting retrieval as a major bottleneck.
• Character-level and self-attention components contribute to performance; ablations show the value of supporting facts and training data diversity.
• Most questions require bridge-entity-based chain reasoning (Type I) or comparison-based reasoning, with a substantial portion needing multiple supporting facts.