[Paper Review] Recurrent One-Hop Predictions for Reasoning over Knowledge Graphs

Wenpeng Yin, Yadollah Yaghoobzadeh+1 more|arXiv (Cornell University)|Jun 12, 2018

Advanced Graph Neural Networks18 references4 citations

TL;DR

This paper proposes ROP (Recurrent One-Hop Predictors), a novel RNN-based framework that models multi-hop knowledge graph (KG) paths by sequentially predicting entities step-by-step using recurrent neural networks. By updating entity and relation embeddings at each hop with training signals, ROP achieves state-of-the-art performance on both multi-hop knowledge base completion and path query answering, significantly improving hits at 10 (H@10) by up to 6% on enhanced datasets.

ABSTRACT

Large scale knowledge graphs (KGs) such as Freebase are generally incomplete. Reasoning over multi-hop (mh) KG paths is thus an important capability that is needed for question answering or other NLP tasks that require knowledge about the world. mh-KG reasoning includes diverse scenarios, e.g., given a head entity and a relation path, predict the tail entity; or given two entities connected by some relation paths, predict the unknown relation between them. We present ROPs, recurrent one-hop predictors, that predict entities at each step of mh-KB paths by using recurrent neural networks and vector representations of entities and relations, with two benefits: (i) modeling mh-paths of arbitrary lengths while updating the entity and relation representations by the training signal at each step; (ii) handling different types of mh-KG reasoning in a unified framework. Our models show state-of-the-art for two important multi-hop KG reasoning tasks: Knowledge Base Completion and Path Query Answering.

Motivation & Objective

• To address the challenge of reasoning over long, arbitrary-length multi-hop paths in incomplete knowledge graphs.
• To unify diverse multi-hop KG reasoning tasks—such as knowledge base completion and path query answering—under a single, end-to-end trainable framework.
• To improve entity and relation representation learning by incorporating training signals at every hop in a path, rather than only at the final prediction.
• To demonstrate the importance of modeling intermediate entities in path query answering, where entity context influences relation prediction.
• To release an enhanced version of the BaseKGP dataset with within-path entities to support future research.

Proposed method

• ROP uses a recurrent neural network (RNN) to process a sequence of relations in a multi-hop KG path, one at a time, to predict the corresponding sequence of intermediate entities.
• At each step, the model takes the current relation embedding and the hidden state from the previous step to predict the next entity, updating the hidden state accordingly.
• Three architectures are proposed: ROP_ARC1, ROP_ARC2, and ROP_ARC3, differing in how the head entity and previous predictions are incorporated into the hidden state for entity prediction.
• ROP_ARC2 improves performance by directly combining the head entity representation with the RNN’s hidden state at each step, ensuring the starting entity remains influential even in long paths.
• ROP_ARC3 further enhances predictions by also incorporating the previously predicted entity’s embedding into the input for the next step, leveraging recent context.
• The model uses GRU or eGRU (enhanced GRU) for better gradient flow and expressivity, outperforming simpler additive (ADD) mechanisms.

Experimental results

Research questions

• RQ1 Can a recurrent neural network effectively model multi-hop KG paths with arbitrary lengths while maintaining strong reasoning performance?
• RQ2 Does incorporating training signals from intermediate entities and relations at each step improve the quality of entity and relation embeddings?
• RQ3 Can a single unified framework handle both multi-hop knowledge base completion and path query answering tasks effectively?
• RQ4 How does the inclusion of intermediate entities affect reasoning performance in path query answering, especially when relations are not symmetric?
• RQ5 To what extent do different RNN architectures (e.g., GRU vs. ADD) and attention mechanisms impact prediction accuracy?

Key findings

• ROP_ARC3 with eGRU achieved a new state-of-the-art H@10 of 56.7% on the enhanced BaseKGP dataset, a 6% improvement over prior baselines.
• ROP_ARC2 (GRU) achieved 54.9% H@10 on BaseKGP and 56.3% on EnhancedKGP, outperforming Comp-TransE by over 4% in H@10.
• The performance drop on even-length paths was strongly correlated (rho > 0.9) with the presence of inverse relations, suggesting a modeling challenge for 1-to-N inverse relations.
• Forwarding the head entity directly into the prediction at each step (ROP_ARC2) outperformed using it only as an initial hidden state (ROP_ARC1), indicating that long-range influence of the head entity is critical.
• Incorporating the previously predicted entity (ROP_ARC3) further improved results, showing that recent context is vital for accurate entity prediction in long paths.
• The eGRU variant consistently outperformed the ADD mechanism, highlighting the importance of gated mechanisms in capturing long-range dependencies in KG reasoning.