[Paper Review] Quantifying Memorization Across Neural Language Models

Nicholas Carlini, Daphne Ippolito+4 more|arXiv (Cornell University)|Feb 15, 2022

Topic ModelingComputer Science156 citations

TL;DR

The paper quantifies how memorization in neural language models scales with model size, data duplication, and context length, showing log-linear growth and the impact of deduplication on reducing memorization.

ABSTRACT

Large language models (LMs) have been shown to memorize parts of their training data, and when prompted appropriately, they will emit the memorized training data verbatim. This is undesirable because memorization violates privacy (exposing user data), degrades utility (repeated easy-to-memorize text is often low quality), and hurts fairness (some texts are memorized over others). We describe three log-linear relationships that quantify the degree to which LMs emit memorized training data. Memorization significantly grows as we increase (1) the capacity of a model, (2) the number of times an example has been duplicated, and (3) the number of tokens of context used to prompt the model. Surprisingly, we find the situation becomes more complicated when generalizing these results across model families. On the whole, we find that memorization in LMs is more prevalent than previously believed and will likely get worse as models continues to scale, at least without active mitigations.

Motivation & Objective

• Quantify the extent of memorization across different model families and datasets using a concrete extractability definition.
• Characterize how memorization scales with model size, data duplication, and prompt length.
• Investigate how decoding strategies and evaluation data sampling affect memorization measurements.
• Assess whether data deduplication or dataset modifications mitigate memorization across models.

Proposed method

• Define memorization via extractability: a string s is extractable with k tokens of context if a prefix p exists in training data such that f(p) outputs s (greedy decoding).
• Evaluate memorization by prompting prefixes from training data and measuring exact suffix reproduction across subsets of sequences.
• Use two sampling schemes: (i) uniformly random data subset, and (ii) duplicate-normalized sampling across lengths and duplication counts to stress-test memorization.
• Prompt models with varying context lengths (50 to 450+ tokens) to study discoverability of memorized sequences.
• Replicate experiments across model families (GPT-Neo on The Pile; T5 on C4; OPT on The Pile) to test generality of scaling trends.
• Compare greedy decoding with beam search to assess decoding’s effect on extractable memorization.

Experimental results

Research questions

• RQ1 How does memorization scale with model size within a model family?
• RQ2 How does data duplication (frequency of repeated examples) influence memorization?
• RQ3 How does the amount of context (prompt length) affect the discoverability of memorized data?
• RQ4 Do deduplicated training datasets reduce memorization across models and tasks?
• RQ5 How do alternative model families and training datasets impact the observed memorization scaling laws?

Key findings

• Memorization grows log-linearly with model size within a family; doubling parameters yields a sizable increase in extractable data.
• More duplicated training examples lead to higher memorization, with a clear log-linear trend across duplication counts.
• Longer context tokens greatly increase extractability, demonstrating a discoverability phenomenon where memorization becomes apparent only with sufficient context.
• Beam search slightly increases extractable memorization compared to greedy decoding, but the effect is modest.
• Deduplicated training data reduces memorization for low to moderate repetition levels, though high-repetition cases show diminished mitigation efficacy.
• Across model families (GPT-Neo on The Pile, T5 on C4, OPT on The Pile), scale effects persist but magnitudes differ, suggesting data curation can influence memorization extent.