[Paper Review] Learning to Exploit Long-term Relational Dependencies in Knowledge Graphs

Lingbing Guo, Zequn Sun+1 more|arXiv (Cornell University)|May 13, 2019

Advanced Graph Neural Networks106 citations

TL;DR

RSNs learn from relational paths with a skipping mechanism to capture long-term dependencies in knowledge graphs, enabling improved cross-KG entity alignment and competitive KG completion.

ABSTRACT

We study the problem of knowledge graph (KG) embedding. A widely-established assumption to this problem is that similar entities are likely to have similar relational roles. However, existing related methods derive KG embeddings mainly based on triple-level learning, which lack the capability of capturing long-term relational dependencies of entities. Moreover, triple-level learning is insufficient for the propagation of semantic information among entities, especially for the case of cross-KG embedding. In this paper, we propose recurrent skipping networks (RSNs), which employ a skipping mechanism to bridge the gaps between entities. RSNs integrate recurrent neural networks (RNNs) with residual learning to efficiently capture the long-term relational dependencies within and between KGs. We design an end-to-end framework to support RSNs on different tasks. Our experimental results showed that RSNs outperformed state-of-the-art embedding-based methods for entity alignment and achieved competitive performance for KG completion.

Motivation & Objective

• Motivate KG embedding beyond triple-level learning to capture long-term relational dependencies.
• Develop a path-level embedding method that leverages relational paths rather than only 1-hop neighbors.
• Propose recurrent skipping networks (RSNs) that use a skipping/ residual mechanism to emphasize subject entities in path modeling.
• Design an end-to-end framework for RSNs including biased path sampling and type-based negative sampling.
• Demonstrate empirically that RSNs outperform state-of-the-art methods on entity alignment and are competitive for KG completion.

Proposed method

• Introduce relational-path based learning and RSNs that apply a residual-like skipping to incorporate subject entities when predicting object entities.
• Model relational paths with RSNs by differentiating entities and relations and applying a skip from entity inputs to object predictions.
• Employ biased random walks to sample deep and cross-KG relational paths for training.
• Use type-based noise-contrastive estimation (NCE) to efficiently train RSNs with negative samples derived from entity or relation vocabularies.
• Add reverse relations to enhance connectivity and enable cross-KG path propagation.

Experimental results

Research questions

• RQ1 Can relational-path based embeddings with RSNs surpass triple-based KG embeddings on entity alignment tasks across multiple KGs?
• RQ2 Do biased depth-aware and cross-KG biased random walks improve the propagation of alignment information and embedding quality?
• RQ3 Are RSNs competitive with state-of-the-art KG completion models when learning from relational paths?
• RQ4 Does the path-based approach provide advantages for long-tail entities and cross-domain KG integration?

Key findings

• RSNs outperform state-of-the-art embedding-based methods on entity alignment benchmarks.
• Biased random walks across KGs improve alignment performance beyond RSNs without biases.
• RSNs achieve competitive KG completion results, often exceeding translational models and approaching specialized KG completion models.
• RSNs demonstrate faster convergence and better optimization behavior than standard RNNs/RRNs for relational path modeling.
• Path-based learning with RSNs shows larger gains in Hits@1 and MRR, indicating stronger top-ranked predictions.