[Paper Review] Language Models as Knowledge Bases?

Fabio Petroni, Tim Rocktäschel+5 more|arXiv (Cornell University)|Sep 3, 2019

Topic Modeling27 references113 citations

TL;DR

The paper analyzes how much factual and commonsense knowledge is stored in pretrained language models (BERT, ELMo, etc.) without fine-tuning, comparing them to symbolic KBs and open-domain QA baselines using the LAMA probe across multiple knowledge sources.

ABSTRACT

Recent progress in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. Whilst learning linguistic knowledge, these models may also be storing relational knowledge present in the training data, and may be able to answer queries structured as "fill-in-the-blank" cloze statements. Language models have many advantages over structured knowledge bases: they require no schema engineering, allow practitioners to query about an open class of relations, are easy to extend to more data, and require no human supervision to train. We present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-the-art pretrained language models. We find that (i) without fine-tuning, BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge, (ii) BERT also does remarkably well on open-domain question answering against a supervised baseline, and (iii) certain types of factual knowledge are learned much more readily than others by standard language model pretraining approaches. The surprisingly strong ability of these models to recall factual knowledge without any fine-tuning demonstrates their potential as unsupervised open-domain QA systems. The code to reproduce our analysis is available at https://github.com/facebookresearch/LAMA.

Motivation & Objective

• Assess the extent of relational knowledge stored in large pretrained language models without fine-tuning.
• Compare BERT, ELMo, and other models against symbolic KBs and QA baselines across multiple knowledge sources.
• Identify which types of knowledge (entity relations, commonsense, QA) are learned most readily by pretraining.
• Evaluate open-domain QA capabilities of language models relative to supervised baselines.

Proposed method

• Introduce the LAMA (LAnguage Model Analysis) probe to test factual and commonsense knowledge.
• Construct knowledge sources (Google-RE, T-REx, ConceptNet, SQuAD) and convert facts to cloze templates for model querying.
• Evaluate multiple pretrained models (fairseq-fconv, Transformer-XL, ELMo variants, BERT-base, BERT-large) using a unified 21K-token vocabulary.
• Use rank-based (P@k) metrics and account for one-to-many relations by pruning valid objects from candidates during testing.
• Compare against baselines such as frequency, a relation extraction (RE) system with and without oracle linking, and DrQA open-domain QA.

Experimental results

Research questions

• RQ1 How much relational and commonsense knowledge do pretrained language models store without fine-tuning?
• RQ2 How does model size and architecture (BERT-large vs. BERT-base vs. ELMo variants) affect knowledge recall across knowledge sources?
• RQ3 How does the knowledge retrieved by LMs compare to symbolic KBs and open-domain QA baselines?
• RQ4 Are certain relation types (1-to-1 vs. N-to-M) better captured by pretrained models?
• RQ5 Can open-domain QA performance of LMs approach supervised systems without fine-tuning?

Key findings

• BERT-large and BERT-base outperform other models and sometimes rival oracle-based knowledge extraction on Google-RE and T-REx tasks.
• Factual knowledge recall is strong for some relation types (especially 1-to-1) but weaker for N-to-M relations.
• BERT-large excels in open-domain cloze QA, achieving 57.1% precision@10 vs. 63.5% for a supervised DrQA system when evaluated at P@10 (gap is small for open-domain QA).
• ELMo-5.5B and BERT variants show robustness to query phrasing, but performance correlates with training data exposure (e.g., object mentions in training data).
• Overall, pretrained LMs store substantial relational and commonsense knowledge, enabling near-KB performance without explicit fine-tuning or retrieval pipelines.