[Paper Review] Embedding Entities and Relations for Learning and Inference in Knowledge Bases

Bishan Yang, Wen-tau Yih+3 more|arXiv (Cornell University)|Dec 20, 2014

Topic Modeling37 references2,032 citations

TL;DR

This paper proposes a unified neural-embedding framework for knowledge base representation learning, demonstrating that a simple bilinear model achieves state-of-the-art performance on link prediction (73.2% top-10 accuracy on Freebase), outperforming TransE. It further introduces a novel rule mining method using learned relation embeddings that effectively captures compositional semantics via matrix multiplication, surpassing AMIE on Horn rule extraction involving relational composition.

ABSTRACT

Compared to black-box neural networks, logic rules express explicit knowledge, can provide human-understandable explanations for reasoning processes, and have found their wide application in knowledge graphs and other downstream tasks. As extracting rules manually from large knowledge graphs is labour-intensive and often infeasible, automated rule learning has recently attracted significant interest, and a number of approaches to rule learning for knowledge graphs have been proposed. This survey aims to provide a review of approaches and a classification of state-of-the-art systems for learning first-order logic rules over knowledge graphs. A comparative analysis of various approaches to rule learning is conducted based on rule language biases, underlying methods, and evaluation metrics. The approaches we consider include inductive logic programming (ILP)-based, statistical path generalisation, and neuro-symbolic methods. Moreover, we highlight important and promising application scenarios of rule learning, such as rule-based knowledge graph completion, fact checking, and applications in other research areas.

Motivation & Objective

• To unify existing multi-relational embedding models under a single neural network framework for systematic comparison.
• To evaluate the impact of different entity and relation representation designs on link prediction performance.
• To develop and validate a novel method for mining logical rules from learned embeddings without relying on explicit logical supervision.
• To investigate whether learned embeddings can capture compositional semantics of relations, especially through matrix multiplication.
• To compare the effectiveness of embedding-based rule mining against state-of-the-art confidence-based rule mining (AMIE) on compositional reasoning tasks.

Proposed method

• Formalizes multi-relational learning as a neural network framework where entities are represented as low-dimensional vectors and relations as bilinear or linear operators.
• Uses a bilinear formulation where the score of a triple (e1, r, e2) is computed as e1^T * R_r * e2, with R_r being a learnable matrix for relation r.
• Employs a margin-based ranking loss for training, optimizing for correct triples to have higher scores than incorrect ones.
• Applies t-SNE visualization to analyze the geometric structure of learned relation embeddings.
• Develops EMBEDRULE, a rule mining method that computes the composition of relation embeddings via matrix multiplication and retrieves rules based on nearest-neighbor search in embedding space.
• Uses precision on manually labeled top predictions as evaluation metric for rule mining, with rules filtered for dynamic relations.

Experimental results

Research questions

• RQ1 Can a unified framework generalize existing multi-relational embedding models like NTN and TransE?
• RQ2 Does a simple bilinear formulation outperform existing models like TransE on the link prediction task?
• RQ3 Can learned relation embeddings effectively capture compositional semantics of relations, such as the composition of 'BornInCity' and 'CityInCountry' to infer 'Nationality'?
• RQ4 Can embedding-based rule mining outperform confidence-based rule mining (AMIE) on rules requiring compositional reasoning?
• RQ5 How do different relation representation types (bilinear, additive, diagonal) affect the quality of mined logical rules?

Key findings

• The bilinear model achieves a top-10 accuracy of 73.2% on Freebase link prediction, significantly outperforming TransE’s 54.7%.
• The bilinear formulation consistently outperforms DISTMULT and DISTADD in mining length-2 and length-3 Horn rules, especially as rule length increases.
• EMBEDRULE using bilinear embeddings successfully extracts rules like 'BornInCity ^ CityInCountry → Nationality' purely from relation embeddings, without relying on entity instances.
• DISTMULT and BILINEAR outperform DISTADD in rule mining, indicating that multiplicative composition via matrix multiplication better captures relational semantics than additive composition.
• Visualization shows that DISTMULT embeddings form interpretable clusters (e.g., /film/release_region near /film/country), while DISTADD embeddings lack clear structure.
• The method based on bilinear embeddings with t-SNE initialization (DISTMULT-TANH-EV-INIT) performs best overall, especially for longer rules, suggesting improved modeling of complex semantics.