[Paper Review] Calibrating Factual Knowledge in Pretrained Language Models

Qingxiu Dong, Damai Dai+4 more|arXiv (Cornell University)|Oct 7, 2022

Topic Modeling4 citations

TL;DR

This paper proposes CaliNet, a lightweight, task-agnostic method to calibrate factual knowledge in pretrained language models (PLMs) without retraining. It detects incorrect facts using a contrastive knowledge assessment (CKA) and corrects them by injecting calibrated feed-forward networks with memory slots, achieving high accuracy and generalization on probing and question-answering tasks.

ABSTRACT

Previous literature has proved that Pretrained Language Models (PLMs) can store factual knowledge. However, we find that facts stored in the PLMs are not always correct. It motivates us to explore a fundamental question: How do we calibrate factual knowledge in PLMs without re-training from scratch? In this work, we propose a simple and lightweight method CaliNet to achieve this goal. To be specific, we first detect whether PLMs can learn the right facts via a contrastive score between right and fake facts. If not, we then use a lightweight method to add and adapt new parameters to specific factual texts. Experiments on the knowledge probing task show the calibration effectiveness and efficiency. In addition, through closed-book question answering, we find that the calibrated PLM possesses knowledge generalization ability after fine-tuning. Beyond the calibration performance, we further investigate and visualize the knowledge calibration mechanism.

Motivation & Objective

• To address the problem of incorrect factual knowledge stored in pretrained language models (PLMs), which impairs performance on knowledge-intensive tasks.
• To develop a method that detects and corrects factual errors in PLMs without retraining from scratch.
• To ensure calibrated knowledge generalizes across diverse surface forms of the same fact.
• To investigate the internal mechanism of knowledge calibration at the neuron level.

Proposed method

• Proposes Contrastive Knowledge Assessment (CKA), which evaluates PLMs by comparing prediction scores for a correct fact versus multiple incorrect facts in a contrastive setup.
• Uses cloze-style prompts to transform factual triplets into natural language inputs for probing.
• Introduces CaliNet, a parameter-efficient method that extends specific feed-forward networks (FFNs) in PLMs with calibration memory slots to store and adapt correct facts.
• Injects new parameters into the FFN layers without modifying original PLM weights, enabling knowledge calibration through fine-tuning.
• Employs paraphrased natural sentences as calibration signals to leverage PLMs' semantic modeling capacity.
• Traces model prediction evolution to analyze how and where calibration occurs within the network.

Experimental results

Research questions

• RQ1 How can factual knowledge in PLMs be reliably assessed for correctness, especially when traditional rank-based metrics fail?
• RQ2 Can a lightweight, parameter-efficient method correct factual errors in PLMs without retraining?
• RQ3 Does knowledge calibration via CaliNet generalize to unseen surface forms of the same fact?
• RQ4 What is the internal mechanism of knowledge calibration at the neuron level, and how do calibration memory slots contribute?

Key findings

• CKA reveals that nearly 50% of facts sampled from the T-REx dataset are stored incorrectly in PLMs, highlighting the prevalence of factual errors.
• CaliNet reduces the false positive rate by 35.61% on real human-annotated test data, demonstrating strong calibration effectiveness.
• The calibrated PLM generalizes well to closed-book question answering, outperforming the original model on downstream knowledge-intensive tasks.
• Calibration memory slots are found to be active and selective in correcting specific factual errors, indicating targeted knowledge adaptation.
• The method preserves original model behavior on non-targeted facts, showing minimal catastrophic forgetting.
• Visualization of prediction evolution confirms that calibration occurs primarily in the FFN layers where factual knowledge is stored.