[Paper Review] ProjE: Embedding Projection for Knowledge Graph Completion

Baoxu Shi, Tim Weninger|arXiv (Cornell University)|Nov 16, 2016

Advanced Graph Neural Networks34 references159 citations

TL;DR

ProjE introduces a self-contained, two-layer neural model that projects candidate entities onto a shared target vector to rank them for knowledge graph completion, achieving strong results with a compact parameter size and without heavy feature engineering.

ABSTRACT

With the large volume of new information created every day, determining the validity of information in a knowledge graph and filling in its missing parts are crucial tasks for many researchers and practitioners. To address this challenge, a number of knowledge graph completion methods have been developed using low-dimensional graph embeddings. Although researchers continue to improve these models using an increasingly complex feature space, we show that simple changes in the architecture of the underlying model can outperform state-of-the-art models without the need for complex feature engineering. In this work, we present a shared variable neural network model called ProjE that fills-in missing information in a knowledge graph by learning joint embeddings of the knowledge graph's entities and edges, and through subtle, but important, changes to the standard loss function. In doing so, ProjE has a parameter size that is smaller than 11 out of 15 existing methods while performing $37\%$ better than the current-best method on standard datasets. We also show, via a new fact checking task, that ProjE is capable of accurately determining the veracity of many declarative statements.

Motivation & Objective

• Motivate knowledge graph completion (KGC) as a ranking problem and improve accuracy with a simple, self-contained model.
• Show that small architectural changes can outperform state-of-the-art models without complex feature engineering.
• Demonstrate that a two-layer ProjE with a diagonal combination operator achieves strong performance and scalability.
• Evaluate ProjE on entity and relationship prediction and on a fact-checking task to assess veracity of declarative statements.

Proposed method

• Represent entities and relationships with low-dimensional embeddings.
• Compute a combined input vector using a diagonal, learnable combination operator that merges entity and relation embeddings.
• Project candidate entities onto a shared target vector to obtain a ranking score for each candidate.
• Provide two loss variants: ProjE_pointwise (binary/multiclass style) and ProjE_listwise (joint ranking with softmax), plus ProjE_wlistwise (weighted listwise).
• Use candidate sampling (negative sampling inspired by Word2Vec) to manage large candidate sets during training.
• Train with Adam optimizer, L1 regularization, and dropout to improve generalization.

Experimental results

Research questions

• RQ1 Can ProjE improve KGC performance with a compact, self-contained architecture versus larger, more complex models?
• RQ2 How do pointwise and listwise training objectives compare for ranking candidate entities in KGC?
• RQ3 Does projection onto a shared target vector effectively capture h-r-t interactions for both entity and relation prediction?
• RQ4 Can ProjE perform well on real-world fact checking by using path-based inputs and minimal prerequisites?

Key findings

• ProjE variants outperform many existing methods on entity prediction and relationship prediction on FB15K.
• ProjE_wlistwise and ProjE_listwise typically outperform ProjE_pointwise across metrics.
• ProjE uses fewer parameters than many competing models while achieving strong accuracy.
• Candidate sampling rates as low as 25% yield robust performance and help scalability.
• In fact-checking tasks, ProjE achieves higher AUC scores than several baseline methods across DBPedia and SemMedDB datasets.