[Paper Review] Language Models are Open Knowledge Graphs

Chenguang Wang, Xiao Liu+1 more|arXiv (Cornell University)|Oct 22, 2020

Topic ModelingComputer Science47 references81 citations

TL;DR

MaMa unsupervisedly constructs open knowledge graphs from pre-trained language models with a single forward pass, matching corpus facts to LM knowledge and mapping them to fixed and open schemas; it outperforms open IE baselines on TAC KBP and Wikidata tasks.

ABSTRACT

This paper shows how to construct knowledge graphs (KGs) from pre-trained language models (e.g., BERT, GPT-2/3), without human supervision. Popular KGs (e.g, Wikidata, NELL) are built in either a supervised or semi-supervised manner, requiring humans to create knowledge. Recent deep language models automatically acquire knowledge from large-scale corpora via pre-training. The stored knowledge has enabled the language models to improve downstream NLP tasks, e.g., answering questions, and writing code and articles. In this paper, we propose an unsupervised method to cast the knowledge contained within language models into KGs. We show that KGs are constructed with a single forward pass of the pre-trained language models (without fine-tuning) over the corpora. We demonstrate the quality of the constructed KGs by comparing to two KGs (Wikidata, TAC KBP) created by humans. Our KGs also provide open factual knowledge that is new in the existing KGs. Our code and KGs will be made publicly available.

Motivation & Objective

• Show that knowledge stored in pre-trained language models can be directly recovered to build knowledge graphs without human supervision.
• Propose MaMa, a two-stage Match-and-Map pipeline to construct open KGs from text and LM knowledge.
• Evaluate the quality of mapped facts against oracle KGs (TAC KBP and Wikidata) and analyze unmapped (open-schema) facts.

Proposed method

• Represent facts as (head, relation, tail) triples and generate candidate facts by matching LM-stored knowledge with corpus facts using attention weights from a forward LM pass.
• Use beam search over attention matrices to extract best head-tail candidate facts without fine-tuning the LM.
• Filter candidate facts using degree-based, distinct-relations-frequency, and contiguity constraints.
• Map candidate facts to a fixed KG schema via unsupervised entity linking and relation mapping; unmapped facts form an open schema in an open KG.
• Produce an open KG that combines mapped facts in the fixed schema with unmapped facts in the open schema.
• Evaluate mapped facts against oracle KGs (TAC KBP 2013, Wikidata) and analyze unmapped facts for quality and error sources.

Experimental results

Research questions

• RQ1 Can pre-trained language models store and reveal factual knowledge sufficient to construct a KG without supervision?
• RQ2 How well can an unsupervised Match-and-Map pipeline recover facts into existing KG schemas and extend into an open schema?
• RQ3 What is the quality of the open-schema facts and what are the main error sources in entity linking and relation mapping?

Key findings

• MaMa achieves competitive precision (>60%) for mapped facts across TAC KBP and Wikidata without supervision.
• On TAC KBP, MaMa variants reach F1 up to 29.72 with XL models, outperforming Stanford OpenIE by more than 2.6 points in F1.
• On Wikidata, MaMa-XL (GPT-2) attains 22.39 F1, surpassing Stanford OpenIE (16.77).
• Larger models (BERT-LARGE, GPT-2-XL) yield higher-quality KGs, indicating more knowledge is stored in deeper/larger LMs.
• Unmapped facts in the open schema show 35.3% true in Wikidata samples, with 83.2% of true unmapped facts being partially unmapped; remaining errors largely stem from entity linking and missing relation mappings.
• Major error sources include spaCy noun chunk-based entity detection (9.1% errors due to linking) and missing relation mappings (4.5%), with some uninformative relation phrases (open IE-type errors).