[Paper Review] CoKE: Contextualized Knowledge Graph Embedding

Quan Wang, Pingping Huang+7 more|arXiv (Cornell University)|Nov 6, 2019

Advanced Graph Neural Networks42 references69 citations

TL;DR

CoKE learns dynamic, context-aware embeddings for KG entities and relations by encoding graph edges and paths with a Transformer, achieving state-of-the-art results on link prediction and path query answering.

ABSTRACT

Knowledge graph embedding, which projects symbolic entities and relations into continuous vector spaces, is gaining increasing attention. Previous methods allow a single static embedding for each entity or relation, ignoring their intrinsic contextual nature, i.e., entities and relations may appear in different graph contexts, and accordingly, exhibit different properties. This work presents Contextualized Knowledge Graph Embedding (CoKE), a novel paradigm that takes into account such contextual nature, and learns dynamic, flexible, and fully contextualized entity and relation embeddings. Two types of graph contexts are studied: edges and paths, both formulated as sequences of entities and relations. CoKE takes a sequence as input and uses a Transformer encoder to obtain contextualized representations. These representations are hence naturally adaptive to the input, capturing contextual meanings of entities and relations therein. Evaluation on a wide variety of public benchmarks verifies the superiority of CoKE in link prediction and path query answering. It performs consistently better than, or at least equally well as current state-of-the-art in almost every case, in particular offering an absolute improvement of 21.0% in H@10 on path query answering. Our code is available at \url{https://github.com/PaddlePaddle/Research/tree/master/KG/CoKE}.

Motivation & Objective

• Motivate modeling contextual meaning of KG entities and relations beyond static embeddings.
• Introduce a contextualized KG embedding paradigm that treats edges and paths as inputs.
• Leverage Transformer encoders to obtain context-specific representations for KG components.
• Train with masked entity prediction to learn dynamic embeddings adaptable to input contexts.
• Demonstrate state-of-the-art performance on link prediction and path query answering benchmarks.

Proposed method

• Represent graph contexts as sequences (edges s->r->o or paths s->r1->...->rk->o).
• Encode sequences with a stack of Transformer blocks to obtain contextualized embeddings for each element.
• Mask the first or last element to create a predictive task for missing entities.
• Use a shared entity embedding matrix to map predictions to entity probabilities.
• Train with cross-entropy loss and label smoothing to predict the missing entity.
• Evaluate using standard link prediction and path query answering protocols, with filtered ranking.

Experimental results

Research questions

• RQ1 Can contextualized, sequence-based KG embeddings outperform static embeddings on link prediction and path query answering?
• RQ2 How do edge- and path-based contexts enable dynamic representations of entities and relations?
• RQ3 Are Transformer-based contextualized embeddings parameter-efficient for KG completion tasks?

Key findings

• CoKE yields competitive to state-of-the-art results on standard benchmarks for link prediction (FB15k, FB15k-237, WN18, WN18RR).
• On FB15k and FB15k-237, CoKE achieves higher MRR and H@k than strong baselines, e.g., MRRs around 0.855 (FB15k) and 0.364 (FB15k-237).
• On WN18RR, CoKE matches or surpasses baselines with MRR around 0.484 and H@10 around 0.553, while maintaining strong performance on WN18 (MRR ~0.952).
• For path query answering, CoKE delivers up to 21.0% absolute improvement in H@10 over prior best methods on Freebase, and substantial gains on WordNet.
• CoKE remains parameter-efficient relative to comparable models, using a relatively small embedding size (D=256) yet achieving strong results.