[Paper Review] Extracting Training Data from Large Language Models

Nicholas Carlini, Florian Tramèr+10 more|arXiv (Cornell University)|Dec 14, 2020

Privacy-Preserving Technologies in Data61 references274 citations

TL;DR

The paper demonstrates black-box training data extraction attacks on GPT-2, recovering hundreds of verbatim training examples and revealing sensitive information, and analyzes factors affecting memorization with recommendations for mitigation.

ABSTRACT

It has become common to publish large (billion parameter) language models that have been trained on private datasets. This paper demonstrates that in such settings, an adversary can perform a training data extraction attack to recover individual training examples by querying the language model. We demonstrate our attack on GPT-2, a language model trained on scrapes of the public Internet, and are able to extract hundreds of verbatim text sequences from the model's training data. These extracted examples include (public) personally identifiable information (names, phone numbers, and email addresses), IRC conversations, code, and 128-bit UUIDs. Our attack is possible even though each of the above sequences are included in just one document in the training data. We comprehensively evaluate our extraction attack to understand the factors that contribute to its success. Worryingly, we find that larger models are more vulnerable than smaller models. We conclude by drawing lessons and discussing possible safeguards for training large language models.

Motivation & Objective

• Demonstrate a practical training data extraction attack against large language models using only black-box access.
• Quantify the extent and nature of memorization in GPT-2 when trained on publicly sourced data.
• Identify factors (model size, data frequency) that influence memorization and leakage.
• Propose mitigations and discuss privacy-preserving training approaches and their trade-offs.

Proposed method

• Generate a large set of high-likelihood samples from the target LM using various sampling strategies.
• Rank and select likely memorized sequences via multiple membership-inference metrics against reference models.
• Validate memorized outputs by verifying against the model’s training data with collaboration from model authors.
• Systematically vary model size, sampling configurations, and data characteristics to study memorization behavior.
• Analyze the impact of string frequency, model size, and attack configuration on extraction success.
• Discuss practical mitigations including differential privacy, data deduplication, and other strategies.

Experimental results

Research questions

• RQ1 To what extent does a large language model memorize and verbatimly leak training data under black-box access?
• RQ2 What factors (model size, data frequency, sampling method) influence memorization and extraction success?
• RQ3 How effective are different attack configurations and membership-inference metrics at identifying memorized content?
• RQ4 What mitigations can reduce memorization without severely harming model utility?
• RQ5 Are the observed memorization phenomena likely to generalize to models beyond GPT-2?

Key findings

• An attacker can extract hundreds of verbatim training sequences from GPT-2 using black-box queries.
• In the best configuration, 67% of candidate samples are verbatim training examples.
• Memorized content can include names, emails, phone numbers, IRC conversations, code, and 128-bit UUIDs.
• Larger models show greater vulnerability to memorization than smaller ones within the GPT-2 family.
• Two dozen attack strategies (sampling, temperature decay, internet-prefix conditioning, multiple inference metrics) were evaluated to reveal robust extraction patterns.
• Mitigations such as differential privacy and careful data deduplication can help but do not fully prevent leakage.