[Paper Review] A Review on Language Models as Knowledge Bases

Badr AlKhamissi, Millicent Li+3 more|arXiv (Cornell University)|Apr 12, 2022

Topic Modeling63 citations

TL;DR

This survey analyzes how pretrained language models encode knowledge implicitly and evaluates five KB-like aspects—Access, Editability, Consistency, Reasoning, and Explainability—to determine how LMs can function as knowledge bases. It synthesizes methods to access, update, reason about, and explain LM-derived knowledge and outlines future challenges.

ABSTRACT

Recently, there has been a surge of interest in the NLP community on the use of pretrained Language Models (LMs) as Knowledge Bases (KBs). Researchers have shown that LMs trained on a sufficiently large (web) corpus will encode a significant amount of knowledge implicitly in its parameters. The resulting LM can be probed for different kinds of knowledge and thus acting as a KB. This has a major advantage over traditional KBs in that this method requires no human supervision. In this paper, we present a set of aspects that we deem a LM should have to fully act as a KB, and review the recent literature with respect to those aspects.

Motivation & Objective

• Assess the extent to which pretrained language models encode knowledge implicitly as a knowledge base.
• Identify core aspects (access, editability, consistency, reasoning, explainability) needed for LMs to function as KBs.
• Survey current methods and their limitations for querying, updating, and reasoning with LM knowledge.
• Highlight challenges and propose directions for future research in LMs-as-KBs.

Proposed method

• Define and compare LMs and KBs, outlining five KB-relevant aspects (access, editability, consistency, reasoning, explainability).
• Review literature on probing and querying LMs, including finetuning and prompting as access mechanisms.
• Summarize model editing approaches (finetuning, hyper networks, Rank-One editing) for updating facts.
• Examine reasoning capabilities in LMs (symbolic, mathematical, commonsense) and their limitations.
• Discuss explainability and interpretability methods (probing, attention analysis, causal tracing, influence functions).

Experimental results

Research questions

• RQ1 What aspects are essential for LMs to function as knowledge bases (access, editability, consistency, reasoning, explainability)?
• RQ2 How do current methods enable access to LM-stored knowledge and what are their limitations?
• RQ3 What approaches exist for editing or updating knowledge in LMs, and how do they preserve stability elsewhere?
• RQ4 To what extent can LMs perform reasoning comparable to traditional KBs, and where do they fall short?
• RQ5 How can explainability and interpretability be achieved for LM-based knowledge retrieval and reasoning?

Key findings

• LMs encode substantial knowledge implicitly, but the knowledge is diffused and hard to update or control.
• Accessing LM knowledge effectively remains challenging; prompting and finetuning are prominent methods but have trade-offs.
• Model editing approaches (e.g., knowledge editors, hyper networks, ROME) address localized updates but face scalability and consistency challenges.
• LMs show promise in various reasoning tasks, including commonsense and symbol-based reasoning, yet struggle with consistent multi-step reasoning and verifiable explanations.
• Explainability tools (probing, causal tracing, influence functions) offer pathways to interpret LM behavior, but explanations can be inconsistent and attention signals are not always faithful.