[Paper Review] PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text

Haitian Sun, Tania Bedrax-Weiss+1 more|arXiv (Cornell University)|Apr 21, 2019

Topic Modeling37 references35 citations

TL;DR

PullNet learns how to iteratively retrieve from a knowledge base and a text corpus to answer open-domain questions, using a question-guided subgraph and graph CNNs to perform multi-hop reasoning with weak supervision.

ABSTRACT

We consider open-domain queston answering (QA) where answers are drawn from either a corpus, a knowledge base (KB), or a combination of both of these. We focus on a setting in which a corpus is supplemented with a large but incomplete KB, and on questions that require non-trivial (e.g., ``multi-hop'') reasoning. We describe PullNet, an integrated framework for (1) learning what to retrieve (from the KB and/or corpus) and (2) reasoning with this heterogeneous information to find the best answer. PullNet uses an {iterative} process to construct a question-specific subgraph that contains information relevant to the question. In each iteration, a graph convolutional network (graph CNN) is used to identify subgraph nodes that should be expanded using retrieval (or ``pull'') operations on the corpus and/or KB. After the subgraph is complete, a similar graph CNN is used to extract the answer from the subgraph. This retrieve-and-reason process allows us to answer multi-hop questions using large KBs and corpora. PullNet is weakly supervised, requiring question-answer pairs but not gold inference paths. Experimentally PullNet improves over the prior state-of-the art, and in the setting where a corpus is used with incomplete KB these improvements are often dramatic. PullNet is also often superior to prior systems in a KB-only setting or a text-only setting.

Motivation & Objective

• Motivate open-domain QA where answers can come from a KB, a corpus, or both, especially when the KB is incomplete.
• Propose an integrated, iterative retrieval framework that builds a question-specific heterogeneous subgraph.
• Enable end-to-end reasoning over text and KB data using graph CNNs with weak supervision from QA pairs.

Proposed method

• Define a heterogeneous question subgraph consisting of entity, text, and fact nodes.
• Iteratively classify and expand the subgraph by pulling related documents and KB facts using a learned pull-operations classifier.
• Retrieve documents via an IDF-based Lucene pull_docs and facts via a learned similarity S(r,q) between relation embeddings and question encoding.
• Update the subgraph after each iteration to include newly retrieved nodes and edges.
• Use a graph CNN (GRAFT-Net style) to compute node representations and to select which entities to pull and which node is most likely to be the answer.
• Train under weak supervision by leveraging question-answer pairs and shortest-path reasoning signals.

Experimental results

Research questions

• RQ1 Can an iterative, question-guided retrieval process over a KB and a text corpus improve open-domain QA performance when the KB is incomplete?
• RQ2 Does learning what to retrieve (vs. relying on heuristics) yield smaller, more relevant subgraphs and better multi-hop reasoning?
• RQ3 How does combining KB and text sources compare to using either source alone across benchmarks with multi-hop questions?

Key findings

• PullNet achieves state-of-the-art or competitive results on KB+text open-domain QA benchmarks, especially when the KB is incomplete.
• In KB-only settings, PullNet significantly improves 2-hop and 3-hop questions compared to baselines.
• In text-only or incomplete-KB+text settings, PullNet outperforms baselines and often surpasses KB-only or text-only approaches.
• Iterative retrieval yields higher recall with smaller subgraphs than heuristic, one-shot retrieval methods.
• On MetaQA 3-hop questions, PullNet improves hits-at-one from 62.5% to 91.4% in the KB setting.
• In Complex WebQuestions with incomplete KB, PullNet improves over GRAFTNet by 6.8% (absolute) in certain settings.