[Paper Review] What can we learn from Data Leakage and Unlearning for Law?

Jaydeep Borkar|arXiv (Cornell University)|Jul 19, 2023

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper investigates data leakage and unlearning in fine-tuned large language models (LLMs), demonstrating that both pre-training and fine-tuning data—including personally identifiable information (PII)—can be leaked during text generation. It reveals that unlearning high-risk data points can inadvertently increase the vulnerability of previously safe data points, a phenomenon termed the 'onion effect,' raising critical privacy and legal concerns for LLM deployment.

ABSTRACT

Large Language Models (LLMs) have a privacy concern because they memorize training data (including personally identifiable information (PII) like emails and phone numbers) and leak it during inference. A company can train an LLM on its domain-customized data which can potentially also include their users' PII. In order to comply with privacy laws such as the "right to be forgotten", the data points of users that are most vulnerable to extraction could be deleted. We find that once the most vulnerable points are deleted, a new set of points become vulnerable to extraction. So far, little attention has been given to understanding memorization for fine-tuned models. In this work, we also show that not only do fine-tuned models leak their training data but they also leak the pre-training data (and PII) memorized during the pre-training phase. The property of new data points becoming vulnerable to extraction after unlearning and leakage of pre-training data through fine-tuned models can pose significant privacy and legal concerns for companies that use LLMs to offer services. We hope this work will start an interdisciplinary discussion within AI and law communities regarding the need for policies to tackle these issues.

Motivation & Objective

• To investigate whether fine-tuned LLMs can leak data from both their pre-training and fine-tuning datasets during text generation.
• To examine the impact of unlearning on the privacy of remaining data points in the dataset, particularly regarding PII.
• To explore the legal and privacy implications of data leakage in LLMs, especially in the context of the 'right to be forgotten' and data protection regulations.
• To initiate an interdisciplinary discussion between AI and law communities on policy needs for mitigating data leakage and unlearning side effects.

Proposed method

• Fine-tuned GPT-2 on the WikiText-103 and WikiText-2 datasets to evaluate data leakage in domain-specific fine-tuned models.
• Generated 2000 text samples using top-k sampling (k=40) with both start-of-sequence tokens and random Common Crawl tokens as prompts.
• Evaluated memorization by searching for n-grams in generated samples that match the fine-tuning dataset and by performing internet searches to detect pre-training data leakage.
• Measured leakage using perplexity and zlib entropy to rank and sort generated samples by memorization likelihood.
• Performed exact unlearning by removing 44 high-risk email addresses from the WikiText-2 dataset and retraining GPT-2 to assess the impact on remaining data points.
• Analyzed embeddings of leaked and unlearned points using pre-trained glove-wiki-gigaword-50 to identify spatial proximity and similarity patterns.

Figure 1: A potential phone number (that doesn’t belong to Wiki-Text103) being leaked after querying GPT-2 fine-tuned on Wiki-Text103. This indicates that examples memorized during pre-training could be leaked even by the fine-tuned model.

Experimental results

Research questions

• RQ1 Can fine-tuned LLMs leak personally identifiable information (PII) from both their fine-tuning and pre-training datasets during text generation?
• RQ2 What is the effect of unlearning specific high-risk data points on the privacy of other, previously safe data points in the dataset?
• RQ3 To what extent can pre-training data, such as URLs, phone numbers, or location coordinates, be inherited and leaked by fine-tuned models?
• RQ4 How do the structural and embedding characteristics of leaked data points relate to their vulnerability after unlearning?

Key findings

• Fine-tuned GPT-2 models leaked actual PII such as phone numbers, email addresses, URLs, and location coordinates not present in the fine-tuning dataset, indicating inheritance from pre-training data.
• The model leaked 44 email addresses from the Enron dataset embedded in WikiText-2, which were not in the original training data but were found via internet search, confirming pre-training data leakage.
• After unlearning the 44 most vulnerable email addresses, 20 new email addresses—previously safe—were leaked, demonstrating the 'onion effect' where unlearning increases the risk for neighboring data points.
• The leaked onion points had lower perplexity and were spatially close in embedding space to the originally leaked emails, indicating they were similarly vulnerable due to semantic and structural similarity.
• The results show that unlearning can inadvertently compromise the privacy of non-targeted data points, undermining the goal of data minimization and compliance with 'right to be forgotten' policies.

(a) zlib entropy and perplexity of GPT-2