[Paper Review] Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey

Xiaokai Wei, Shen Wang+3 more|arXiv (Cornell University)|Oct 16, 2021

Topic Modeling110 references25 citations

TL;DR

A comprehensive survey of Knowledge Enhanced Pretrained Language Models (KE-PLMs), presenting three taxonomies of knowledge sources, granularities, and applications, with discussion of methods, datasets, applications, challenges, and future directions.

ABSTRACT

Pretrained Language Models (PLM) have established a new paradigm through learning informative contextualized representations on large-scale text corpus. This new paradigm has revolutionized the entire field of natural language processing, and set the new state-of-the-art performance for a wide variety of NLP tasks. However, though PLMs could store certain knowledge/facts from training corpus, their knowledge awareness is still far from satisfactory. To address this issue, integrating knowledge into PLMs have recently become a very active research area and a variety of approaches have been developed. In this paper, we provide a comprehensive survey of the literature on this emerging and fast-growing field - Knowledge Enhanced Pretrained Language Models (KE-PLMs). We introduce three taxonomies to categorize existing work. Besides, we also survey the various NLU and NLG applications on which KE-PLM has demonstrated superior performance over vanilla PLMs. Finally, we discuss challenges that face KE-PLMs and also promising directions for future research.

Motivation & Objective

• Motivate the study of integrating knowledge into pretrained language models to address limitations in knowledge awareness.
• Provide a structured taxonomy of KE-PLMs based on knowledge sources, granularity, and applications.
• Survey influential methods, objectives, and datasets across NLU and NLG tasks.
• Discuss challenges and propose promising directions for future KE-PLM research.

Proposed method

• Introduce three taxonomies to categorize KE-PLMs by knowledge source, granularity, and application.
• Review representative methods and knowledge integration strategies across categories (linguistic, encyclopedia, commonsense, domain-specific).
• Summarize datasets and applications demonstrating KE-PLMs’ performance in NLU and NLG tasks.
• Compare approaches using a synthesis table of methods and characteristics.

Experimental results

Research questions

• RQ1 What knowledge sources have been used to build KE-PLMs (linguistic, encyclopedia, commonsense, domain-specific)?
• RQ2 How is knowledge incorporated at different granularities (text chunks, entities, relations, subgraphs) in KE-PLMs?
• RQ3 Which NLP tasks and applications benefit from KE-PLMs, and what benchmarks/datasets illustrate these gains?
• RQ4 What are the main challenges and future directions for KE-PLMs (efficiency, noise robustness, knowledge selection, etc.)?

Key findings

• KE-PLMs integrate diverse sources (linguistic, encyclopedia, commonsense, domain-specific) to enhance knowledge awareness beyond vanilla PLMs.
• Knowledge is leveraged at multiple granularities, including text chunks, entity-level cues, relation triples, and subgraphs, with corresponding modeling approaches.
• KE-PLMs show improved performance on a range of NLU and NLG tasks such as entity typing, relation classification, QA, commonsense reasoning, and text generation.
• A variety of datasets and benchmarks (e.g., LAMA, commonsense QA, KG-related tasks) are used to evaluate KE-PLMs, illustrating broad applicability.
• The survey highlights challenges (efficiency, noise, data quality) and outlines directions for future work (broader applications, more knowledge sources, robust training, and scalable inference).