[Paper Review] Go for a Walk and Arrive at the Answer: Reasoning Over Paths in Knowledge Bases using Reinforcement Learning

Rajarshi Das, Shehzaad Dhuliawala+6 more|arXiv (Cornell University)|Nov 15, 2017

Topic ModelingComputer Science58 references356 citations

TL;DR

The paper introduces Minerva, a reinforcement learning agent that navigates knowledge graphs to answer query questions by discovering reasoning paths, without precomputed paths, and shows competitive results across multiple datasets.

ABSTRACT

Knowledge bases (KB), both automatically and manually constructed, are often incomplete --- many valid facts can be inferred from the KB by synthesizing existing information. A popular approach to KB completion is to infer new relations by combinatory reasoning over the information found along other paths connecting a pair of entities. Given the enormous size of KBs and the exponential number of paths, previous path-based models have considered only the problem of predicting a missing relation given two entities or evaluating the truth of a proposed triple. Additionally, these methods have traditionally used random paths between fixed entity pairs or more recently learned to pick paths between them. We propose a new algorithm MINERVA, which addresses the much more difficult and practical task of answering questions where the relation is known, but only one entity. Since random walks are impractical in a setting with combinatorially many destinations from a start node, we present a neural reinforcement learning approach which learns how to navigate the graph conditioned on the input query to find predictive paths. Empirically, this approach obtains state-of-the-art results on several datasets, significantly outperforming prior methods.

Motivation & Objective

• Motivate automated reasoning over incomplete knowledge bases and the need for scalable, path-based inference.
• Propose a neural reinforcement learning agent that navigates KB graphs conditioned on the query to find answer nodes.
• Enable variable-length reasoning paths and eliminate precomputed path sets.
• Provide an end-to-end learning framework that trains from scratch with reinforcement learning and evaluates on diverse KB datasets.

Proposed method

• Formulate query answering as a finite-horizon deterministic partially observable Markov decision process on a knowledge graph.
• Define states as (current entity, query head, query relation, true answer) and observations as (current entity, query head, query relation).
• Use a policy network (LSTM-based) that encodes path history and query relation to select outgoing labeled-edge actions at each step.
• Incorporate inverse relations to allow the agent to backtrack and undo wrong steps.
• Train with REINFORCE using a moving-average baseline for variance reduction and add entropy regularization to encourage exploration.
• Support a beam search during inference to rank candidate answers by trajectory probability.

Experimental results

Research questions

• RQ1 Can an RL agent learn to navigate large knowledge graphs to answer queries with unknown tail entities?
• RQ2 Does conditioning path selection on the input query improve path discovery over random-walk baselines?
• RQ3 How does Minerva perform relative to embedding-based and logic-rule-based KB completion methods across small and large datasets?
• RQ4 Can Minerva handle partially structured natural language questions and long reasoning chains?
• RQ5 Is Minerva scalable in inference time and robust to training across diverse KBs?

Key findings

• Minerva achieves competitive results on seven knowledge base datasets, often outperforming several baselines.
• On small datasets Kinship and UMLS, embedding methods outperform Minerva, but Minerva matches or exceeds NeuralLP and NTP in several metrics.
• On large datasets wn18rr, fb15k-237, and nell-995, Minerva generally outperforms path-based baselines and shows competitive performance against embedding methods, particularly on certain metrics.
• Minerva can be extended to partially structured natural language queries, demonstrated on WikiMovies with competitive accuracy.
• The model demonstrates the ability to reason over longer chains and provides efficient inference by exploring a local graph neighborhood rather than scoring all entities.