[Paper Review] Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data

Xinyi Wang, Antonis Antoniades+5 more|arXiv (Cornell University)|Jul 20, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes a scalable n-gram pair analysis to trace large language models' capabilities back to their pretraining data, using semantic similarity and frequency counts. It finds that larger models generalize better by leveraging task-relevant n-gram pairs, while smaller models memorize; instruction tuning enhances data utilization without introducing new capabilities.

ABSTRACT

The impressive capabilities of large language models (LLMs) have sparked debate over whether these models genuinely generalize to unseen tasks or predominantly rely on memorizing vast amounts of pretraining data. To explore this issue, we introduce an extended concept of memorization, distributional memorization, which measures the correlation between the LLM output probabilities and the pretraining data frequency. To effectively capture task-specific pretraining data frequency, we propose a novel task-gram language model, which is built by counting the co-occurrence of semantically related $n$-gram pairs from task inputs and outputs in the pretraining corpus. Using the Pythia models trained on the Pile dataset, we evaluate four distinct tasks: machine translation, factual question answering, world knowledge understanding, and math reasoning. Our findings reveal varying levels of memorization, with the strongest effect observed in factual question answering. Furthermore, while model performance improves across all tasks as LLM size increases, only factual question answering shows an increase in memorization, whereas machine translation and reasoning tasks exhibit greater generalization, producing more novel outputs. This study demonstrates that memorization plays a larger role in simpler, knowledge-intensive tasks, while generalization is the key for harder, reasoning-based tasks, providing a scalable method for analyzing large pretraining corpora in greater depth.

Motivation & Objective

• To understand how pretraining data contributes to LLM capabilities beyond exact memorization.
• To investigate the balance between memorization and generalization in large language models at scale.
• To trace model performance back to specific n-gram pairs in the pretraining corpus.
• To evaluate the impact of model size and instruction tuning on data utilization.
• To provide a scalable, interpretable method for analyzing the origins of LLM capabilities.

Proposed method

• Mining semantically similar n-gram pairs from task inputs and outputs using embedding-based similarity (cosine similarity with a pretrained encoder).
• Using the WIMBD framework to search for these n-gram pairs in the full pretraining corpus (e.g., The Pile) and retrieve their counts and source documents.
• Modeling the pretraining data distribution via n-gram pair frequencies, treating them as weak supervision for task-relevant data.
• Approximating the conditional distribution P(y|x) using n-gram frequencies and comparing it to the LM’s predicted distribution via log-probability regression.
• Measuring distribution similarity between pretraining data and LM outputs using R² scores of linear regression between log data and log LM probabilities.
• Analyzing gradient similarity and novelty of generated n-grams to assess memorization vs. generalization.

Experimental results

Research questions

• RQ1 How do task-relevant n-gram pairs in pretraining data relate to model performance across different model sizes?
• RQ2 To what extent do larger models generalize rather than memorize, and how does this relate to pretraining data frequency?
• RQ3 How does instruction tuning affect a model’s reliance on pretraining data, independent of the tuning data itself?
• RQ4 Can emergent abilities be explained by a shift from memorization to generalization driven by sufficient task-relevant pretraining data?
• RQ5 What is the role of n-gram pair frequency in determining model performance and distributional alignment?

Key findings

• Task-relevant n-gram pairs are more representative of task-related data than single n-grams, and their frequency correlates positively with model performance.
• As model size increases, memorization decreases and generalization strengthens, with R² scores of distribution similarity first decreasing then slightly increasing for larger models.
• Emergent abilities in larger models correlate with sufficient task-relevant pretraining data, suggesting a transition from memorization to generalization.
• Small LMs tend to memorize, while large LMs generalize, generating more novel n-grams not seen in pretraining.
• Instruction tuning improves data utilization without memorizing the tuning data, indicating it enhances preexisting capabilities learned during pretraining.
• The distribution similarity between LM outputs and pretraining data decreases with model size, indicating reduced memorization and stronger generalization.