[Paper Review] KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation

Xiaozhi Wang, Tianyu Gao+5 more|arXiv (Cornell University)|Nov 13, 2019

Topic Modeling66 references80 citations

TL;DR

KEPLER unifies knowledge embedding with pre-trained language models by encoding entity descriptions through a PLM and jointly optimizing knowledge embedding and masked language modeling objectives, enabling inductive knowledge embedding and improved NLP performance.

ABSTRACT

Pre-trained language representation models (PLMs) cannot well capture factual knowledge from text. In contrast, knowledge embedding (KE) methods can effectively represent the relational facts in knowledge graphs (KGs) with informative entity embeddings, but conventional KE models cannot take full advantage of the abundant textual information. In this paper, we propose a unified model for Knowledge Embedding and Pre-trained LanguagE Representation (KEPLER), which can not only better integrate factual knowledge into PLMs but also produce effective text-enhanced KE with the strong PLMs. In KEPLER, we encode textual entity descriptions with a PLM as their embeddings, and then jointly optimize the KE and language modeling objectives. Experimental results show that KEPLER achieves state-of-the-art performances on various NLP tasks, and also works remarkably well as an inductive KE model on KG link prediction. Furthermore, for pre-training and evaluating KEPLER, we construct Wikidata5M, a large-scale KG dataset with aligned entity descriptions, and benchmark state-of-the-art KE methods on it. It shall serve as a new KE benchmark and facilitate the research on large KG, inductive KE, and KG with text. The source code can be obtained from https://github.com/THU-KEG/KEPLER.

Motivation & Objective

• Bridge the gap between knowledge graphs and pre-trained language models by jointly modeling knowledge embedding and language modeling.
• Leverage entity descriptions to produce text-enhanced knowledge embeddings.
• Enable inductive knowledge embedding for unseen entities through descriptive text.
• Maintain strong language understanding while integrating factual knowledge without inference overhead beyond standard PLMs.

Proposed method

• Encode textual entity descriptions with a shared PLM to obtain entity embeddings.
• Jointly optimize a knowledge embedding (KE) objective and a masked language modeling (MLM) objective in a single multi-task framework.
• Explore KE variants including entity descriptions as embeddings, entity and relation descriptions as embeddings, and entity embeddings conditioned on relations.
• Use a TransE-like scoring function for KE with negative sampling to train entity and relation representations.
• Initialize from RoBERTa base and pre-train on Wikipedia and BookCorpus with KE objective applied to first 512 tokens of entity descriptions.

Experimental results

Research questions

• RQ1 Can a unified model jointly learn knowledge embedding and language modeling to improve both KG tasks and NLP tasks?
• RQ2 How does encoding entity descriptions as embeddings influence inductive knowledge embedding and link prediction?
• RQ3 What are the performance gains of KEPLER across NLP benchmarks and KE benchmarks compared to existing baselines?
• RQ4 Does the MLM objective remain important for language understanding when KE objectives drive pre-training?
• RQ5 How does KEPLER perform on large-scale KG data and in inductive settings with unseen entities?

Key findings

• KEPLER variants, especially KEPLER-Wiki, achieve strong performance across NLP tasks compared to baselines on TACRED, FewRel, OpenEntity, and GLUE.
• KEPLER demonstrates competitive to state-of-the-art results and shows notable gains in relation classification and entity typing when incorporating KG-derived knowledge.
• KEPLER excels as a KE model in transductive and inductive link prediction on Wikidata5M, highlighting effective inductive capabilities for unseen entities.
• Wikidata5M is introduced as a large-scale KG dataset with aligned entity descriptions and an inductive split to benchmark large KG pre-training and inductive KE.
• In GLUE, KEPLER-Wiki closely matches RoBERTa, indicating knowledge integration does not severely harm general language understanding; smaller description-only variants underperform, underscoring the importance of diverse MLM data.