[Paper Review] Large Language Models Struggle to Learn Long-Tail Knowledge

Nikhil Kandpal, Haikang Deng+3 more|arXiv (Cornell University)|Nov 15, 2022

Topic Modeling96 citations

TL;DR

The paper investigates how LLM factual knowledge depends on the prevalence of related documents in pre-training data, using entity-linked document counts to show correlations and causal effects, and explores retrieval augmentation as a remedy.

ABSTRACT

The Internet contains a wealth of knowledge -- from the birthdays of historical figures to tutorials on how to code -- all of which may be learned by language models. However, while certain pieces of information are ubiquitous on the web, others appear extremely rarely. In this paper, we study the relationship between the knowledge memorized by large language models and the information in pre-training datasets scraped from the web. In particular, we show that a language model's ability to answer a fact-based question relates to how many documents associated with that question were seen during pre-training. We identify these relevant documents by entity linking pre-training datasets and counting documents that contain the same entities as a given question-answer pair. Our results demonstrate strong correlational and causal relationships between accuracy and relevant document count for numerous question answering datasets (e.g., TriviaQA), pre-training corpora (e.g., ROOTS), and model sizes (e.g., 176B parameters). Moreover, while larger models are better at learning long-tail knowledge, we estimate that today's models must be scaled by many orders of magnitude to reach competitive QA performance on questions with little support in the pre-training data. Finally, we show that retrieval-augmentation can reduce the dependence on relevant pre-training information, presenting a promising approach for capturing the long-tail.

Motivation & Objective

• Investigate how an LM's ability to answer fact-based questions relates to the amount of pre-training data containing the relevant entities.
• Identify relevant pre-training documents via entity linking to quantify knowledge exposure across large corpora.
• Assess whether model size and pre-training data scale explain long-tail knowledge learning.
• Examine retrieval augmentation as a method to reduce dependence on rare pre-training information.

Proposed method

• Build a scalable entity-linking pipeline to map salient question/answer entities to documents in pre-training datasets (The Pile, ROOTS, C4, OpenWebText, Wikipedia).
• Count documents where question and answer entities co-occur to identify ‘relevant documents’ for each QA pair.
• Evaluate open-domain QA models (GPT-Neo, BLOOM, GPT-3) in 4-shot setting against TriviaQA and Natural Questions using Ex-Act match, analyzing accuracy vs. relevant document counts.
• Conduct a counterfactual re-training by removing all relevant documents for a sample of questions to test causality between document count and accuracy.
• Explore scaling effects (model size, data scale) and retrieval augmentation (oracle and BM25 retrieval) on rare facts.

Experimental results

Research questions

• RQ1 How does an LM’s QA accuracy correlate with the number of pre-training documents that are relevant to a given question?
• RQ2 Are the observed correlations causal, i.e., does removing relevant pre-training documents degrade QA performance?
• RQ3 To what extent do model size and pre-training data scale improve learning of long-tail knowledge?
• RQ4 Can retrieval augmentation mitigate dependence on pre-training data for rare facts?
• RQ5 Do alternative lightweight methods for identifying relevant documents explain QA performance as well as the co-occurrence-based approach?

Key findings

• QA accuracy strongly correlates with the count of relevant pre-training documents across datasets and models (e.g., BLOOM-176B on TriviaQA).
• Counterfactual re-training shows that removing relevant documents degrades accuracy on questions with many relevant documents, suggesting a causal link.
• Model size has a strong log-linear relation with rare-fact QA performance, implying enormous parameter counts would be needed to match strong baselines on long-tail questions.
• Retrieval augmentation substantially improves performance, especially on rare questions, and can reduce reliance on pre-training data.
• Oracle retrieval dramatically boosts accuracy on rare instances, while BM25-based retrieval provides improvement with mild residual dependence on document count.