[논문 리뷰] A Review on Language Models as Knowledge Bases

Badr AlKhamissi, Millicent Li외 3명|arXiv (Cornell University)|2022. 04. 12.

Topic Modeling인용 수 63

한 줄 요약

이 설문조사는 사전 학습된 언어 모델이 지식을 암묵적으로 어떻게 인코딩하는지 분석하고, Access, Editability, Consistency, Reasoning, Explainability의 다섯 가지 KB-like 측면을 평가하여 LM이 지식 베이스로 기능할 수 있는지 결정합니다. LM에서 파생된 지식을 접근, 업데이트, 추론 및 설명하는 방법을 합성하고, 향후 도전과제를 개략합니다.

ABSTRACT

Recently, there has been a surge of interest in the NLP community on the use of pretrained Language Models (LMs) as Knowledge Bases (KBs). Researchers have shown that LMs trained on a sufficiently large (web) corpus will encode a significant amount of knowledge implicitly in its parameters. The resulting LM can be probed for different kinds of knowledge and thus acting as a KB. This has a major advantage over traditional KBs in that this method requires no human supervision. In this paper, we present a set of aspects that we deem a LM should have to fully act as a KB, and review the recent literature with respect to those aspects.

연구 동기 및 목표

• Assess the extent to which pretrained language models encode knowledge implicitly as a knowledge base.
• Identify core aspects (access, editability, consistency, reasoning, explainability) needed for LMs to function as KBs.
• Survey current methods and their limitations for querying, updating, and reasoning with LM knowledge.
• Highlight challenges and propose directions for future research in LMs-as-KBs.

제안 방법

• Define and compare LMs and KBs, outlining five KB-relevant aspects (access, editability, consistency, reasoning, explainability).
• Review literature on probing and querying LMs, including finetuning and prompting as access mechanisms.
• Summarize model editing approaches (finetuning, hyper networks, Rank-One editing) for updating facts.
• Examine reasoning capabilities in LMs (symbolic, mathematical, commonsense) and their limitations.
• Discuss explainability and interpretability methods (probing, attention analysis, causal tracing, influence functions).

실험 결과

연구 질문

• RQ1 What aspects are essential for LMs to function as knowledge bases (access, editability, consistency, reasoning, explainability)?
• RQ2 How do current methods enable access to LM-stored knowledge and what are their limitations?
• RQ3 What approaches exist for editing or updating knowledge in LMs, and how do they preserve stability elsewhere?
• RQ4 To what extent can LMs perform reasoning comparable to traditional KBs, and where do they fall short?
• RQ5 How can explainability and interpretability be achieved for LM-based knowledge retrieval and reasoning?

주요 결과

• LMs encode substantial knowledge implicitly, but the knowledge is diffused and hard to update or control.
• Accessing LM knowledge effectively remains challenging; prompting and finetuning are prominent methods but have trade-offs.
• Model editing approaches (e.g., knowledge editors, hyper networks, ROME) address localized updates but face scalability and consistency challenges.
• LMs show promise in various reasoning tasks, including commonsense and symbol-based reasoning, yet struggle with consistent multi-step reasoning and verifiable explanations.
• Explainability tools (probing, causal tracing, influence functions) offer pathways to interpret LM behavior, but explanations can be inconsistent and attention signals are not always faithful.