[Paper Review] Knowledge Graph Embedding: An Overview

Xiou Ge, Yun-Cheng Wang+2 more|arXiv (Cornell University)|Sep 21, 2023

Advanced Graph Neural Networks4 citations

TL;DR

This paper provides a comprehensive overview of knowledge graph embedding (KGE) models for knowledge graph completion, focusing on distance-based and semantic matching-based methods. It introduces CompoundE and CompoundE3D, which unify geometric transformations like translation and rotation to model complex relation patterns, and highlights emerging trends in integrating pre-trained language models (PLMs) with KGE for improved link prediction performance.

ABSTRACT

Many mathematical models have been leveraged to design embeddings for representing Knowledge Graph (KG) entities and relations for link prediction and many downstream tasks. These mathematically-inspired models are not only highly scalable for inference in large KGs, but also have many explainable advantages in modeling different relation patterns that can be validated through both formal proofs and empirical results. In this paper, we make a comprehensive overview of the current state of research in KG completion. In particular, we focus on two main branches of KG embedding (KGE) design: 1) distance-based methods and 2) semantic matching-based methods. We discover the connections between recently proposed models and present an underlying trend that might help researchers invent novel and more effective models. Next, we delve into CompoundE and CompoundE3D, which draw inspiration from 2D and 3D affine operations, respectively. They encompass a broad spectrum of techniques including distance-based and semantic-based methods. We will also discuss an emerging approach for KG completion which leverages pre-trained language models (PLMs) and textual descriptions of entities and relations and offer insights into the integration of KGE embedding methods with PLMs for KG completion.

Motivation & Objective

• To provide a unified, comprehensive overview of existing knowledge graph embedding (KGE) models for knowledge graph completion.
• To identify and analyze key trends in KGE model design, particularly the integration of geometric transformations such as translation, rotation, and scaling.
• To explore the synergy between KGE methods and pre-trained language models (PLMs) for enhanced link prediction and knowledge reasoning.
• To curate essential research resources, including benchmark datasets, public knowledge graphs, and leaderboards, to support future KGE research.
• To inspire novel KGE model development through the introduction of CompoundE and CompoundE3D, which unify diverse KGE techniques via affine operations in 2D and 3D spaces.

Proposed method

• Categorizes KGE models into two main branches: distance-based methods (e.g., TransE, RotatE) and semantic matching-based methods (e.g., ComplEx, SimplE).
• Introduces CompoundE and CompoundE3D as unified frameworks that combine multiple geometric transformations—such as translation, rotation, scaling, and reflection—using 2D and 3D affine operations.
• Analyzes the mathematical structure of KGE models through the lens of geometric transformations, revealing underlying connections between seemingly disparate models.
• Proposes a framework for integrating pre-trained language models (PLMs) with KGE by leveraging textual descriptions of entities and relations, using models like KEPLER, BERTRL, and KGT5.
• Employs contrastive learning and self-supervised objectives in models like SimKGC and StAR to enhance semantic matching between head and tail entity representations.
• Utilizes sequence-to-sequence modeling via T5 in KGT5 to perform end-to-end link prediction and question answering using natural language signals.

Figure 2: Illustration of Knowledge Panel from Google Search.

Experimental results

Research questions

• RQ1 How do geometric transformations such as translation, rotation, and scaling contribute to modeling diverse relation patterns in knowledge graphs?
• RQ2 What underlying structural and mathematical connections exist between seemingly distinct KGE models like TransE, RotatE, and ComplEx?
• RQ3 To what extent can pre-trained language models (PLMs) be effectively integrated with KGE to improve link prediction and reasoning performance?
• RQ4 How do unified frameworks like CompoundE and CompoundE3D generalize and improve upon existing KGE models through combined geometric operations?
• RQ5 What are the most effective self-supervised and contrastive learning strategies for enhancing semantic matching in KGE models using textual descriptions?

Key findings

• CompoundE and CompoundE3D unify a broad spectrum of KGE models by modeling relations through 2D and 3D affine transformations, respectively, demonstrating improved modeling capacity.
• The integration of geometric transformations such as rotation and translation enables effective modeling of symmetric, antisymmetric, and other complex relation patterns.
• Models like RotatE and QuatE achieve state-of-the-art performance on standard benchmarks (e.g., FB15k-237) by modeling relations as rotations in complex vector spaces.
• PLM-based KGE models such as KEPLER and BERTRL show strong performance by initializing embeddings with textual descriptions and fine-tuning via masked language modeling or relation path modeling.
• KGT5 achieves competitive results in both link prediction and question answering by using a T5-based sequence-to-sequence framework with natural language prompts.
• SimKGC and StAR demonstrate that contrastive learning with in-batch and self-negative sampling significantly improves performance on link prediction tasks, especially when combined with graph structural inductive biases.

Figure 3: Connections between different knowledge graph embedding models