[Paper Review] KELM: Knowledge Enhanced Pre-Trained Language Representations with Message Passing on Hierarchical Relational Graphs

Yinquan Lu, H. Lu+2 more|arXiv (Cornell University)|Sep 9, 2021

Topic ModelingComputer Science37 references19 citations

TL;DR

KELM proposes a knowledge-enhanced fine-tuning framework that dynamically integrates factual knowledge from knowledge graphs (KGs) into pre-trained language models like BERT via a unified knowledge-enhanced text graph (UKET). By leveraging hierarchical relational message passing across text, entity mentions, and KG sub-graphs, KELM resolves knowledge ambiguity and achieves state-of-the-art performance on MRC tasks, outperforming prior models by up to +3.2 EM on ReCoRD and +3.1 EM on MultiRC.

ABSTRACT

Incorporating factual knowledge into pre-trained language models (PLM) such as BERT is an emerging trend in recent NLP studies. However, most of the existing methods combine the external knowledge integration module with a modified pre-training loss and re-implement the pre-training process on the large-scale corpus. Re-pretraining these models is usually resource-consuming, and difficult to adapt to another domain with a different knowledge graph (KG). Besides, those works either cannot embed knowledge context dynamically according to textual context or struggle with the knowledge ambiguity issue. In this paper, we propose a novel knowledge-aware language model framework based on fine-tuning process, which equips PLM with a unified knowledge-enhanced text graph that contains both text and multi-relational sub-graphs extracted from KG. We design a hierarchical relational-graph-based message passing mechanism, which can allow the representations of injected KG and text to mutually update each other and can dynamically select ambiguous mentioned entities that share the same text. Our empirical results show that our model can efficiently incorporate world knowledge from KGs into existing language models such as BERT, and achieve significant improvement on the machine reading comprehension (MRC) task compared with other knowledge-enhanced models.

Motivation & Objective

• To address the limitations of existing knowledge-enhanced PLMs that require costly re-pretraining and struggle with knowledge ambiguity.
• To enable dynamic, context-aware knowledge injection during fine-tuning rather than pretraining, improving adaptability across domains.
• To resolve knowledge ambiguity by selecting the most contextually relevant mentioned entities from multi-relational KGs.
• To efficiently fuse heterogeneous textual and structured knowledge using a unified graph representation without retraining the base PLM.

Proposed method

• Constructs a Unified Knowledge-enhanced Text Graph (UKET) integrating three components: text-only sequences, entity mention-to-KG entity links, and injected KG sub-graphs centered on mentioned entities.
• Applies a hierarchical relational message passing mechanism that alternately updates representations across the text graph, entity-link graph, and KG sub-graphs.
• Uses attention-based scoring to dynamically select the most contextually relevant mentioned entity from multiple candidates sharing the same text in the KG.
• Employs a message passing strategy that propagates information across multi-relational KGs, enabling context-sensitive representation of entities.
• Integrates knowledge injection at the fine-tuning stage, avoiding the need for re-pretraining the base PLM on large-scale corpora.

Experimental results

Research questions

• RQ1 Can knowledge injection during fine-tuning outperform re-pretraining-based methods in terms of efficiency and domain adaptability?
• RQ2 How can a model dynamically select the correct entity from multiple ambiguous mentions in a KG based on textual context?
• RQ3 Can hierarchical message passing on a unified text-KG graph improve factual reasoning in machine reading comprehension?
• RQ4 To what extent does dynamic knowledge integration reduce noise from irrelevant or ambiguous entity candidates?

Key findings

• KELM achieves a +3.2 EM and +1.9 F1 improvement over the previous SOTA knowledge-enhanced model on the ReCoRD dataset.
• On MultiRC, KELM improves by +3.1 EM and +0.8 F1 over BERT_large, and gains +1.5 in combined EM+F1 over KT-NET.
• The model effectively resolves knowledge ambiguity by selecting the correct entity sense (e.g., 'ban' as a noun vs. verb) based on context, as shown in case studies.
• The attention-based entity selection mechanism assigns higher weights to contextually relevant entities, such as 'ban.n.04' (0.715) over 'ban.v.02' (0.205) in the given example.
• KELM demonstrates strong generalization across domains without requiring retraining of the base PLM, enabling rapid adaptation to new knowledge graphs.