[Paper Review] Knowledge Unlearning for Mitigating Privacy Risks in Language Models

Joel Jang, Dongkeun Yoon+5 more|arXiv (Cornell University)|Oct 4, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes knowledge unlearning as a post-hoc method to mitigate privacy risks in large language models by performing gradient ascent on memorized sequences, effectively reducing their extractability without retraining. The approach achieves strong privacy protection—often improving model performance—while being significantly more efficient and robust than data preprocessing or differential privacy methods.

ABSTRACT

Pretrained Language Models (LMs) memorize a vast amount of knowledge during initial pretraining, including information that may violate the privacy of personal lives and identities. Previous work addressing privacy issues for language models has mostly focused on data preprocessing and differential privacy methods, both requiring re-training the underlying LM. We propose knowledge unlearning as an alternative method to reduce privacy risks for LMs post hoc. We show that simply performing gradient ascent on target token sequences is effective at forgetting them with little to no degradation of general language modeling performances for larger LMs; it sometimes even substantially improves the underlying LM with just a few iterations. We also find that sequential unlearning is better than trying to unlearn all the data at once and that unlearning is highly dependent on which kind of data (domain) is forgotten. By showing comparisons with a previous data preprocessing method and a decoding method known to mitigate privacy risks for LMs, we show that unlearning can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being much more efficient and robust. We release the code and dataset needed to replicate our results at https://github.com/joeljang/knowledge-unlearning.

Motivation & Objective

• To address the growing risk of privacy violations in large language models due to memorization of sensitive data such as PII and private identities.
• To propose a post-ho training solution that avoids costly retraining when individuals invoke their Right-To-Be-Forgotten (RTBF).
• To develop an efficient, scalable method that enables forgetting specific memorized sequences without degrading general language modeling capabilities.

Proposed method

• Applying gradient ascent on target token sequences to reverse memorization, effectively unlearning them from the model.
• Using a novel extraction likelihood (EL) metric to quantify how likely a target sequence is to be extracted, varying prefix lengths to simulate attack strength.
• Performing sequential unlearning by forgetting chunks of data one at a time, rather than all at once, to improve effectiveness.
• Evaluating unlearning on GPT-Neo models (125M, 1.3B, 2.7B) across 9 NLP benchmarks and 4 dialogue tasks to measure performance stability.
• Comparing the method empirically with data deduplication and differential privacy decoding, showing superior efficiency and robustness.
• Defining a forgetting threshold based on EL values to determine when a sequence is effectively forgotten.

Experimental results

Research questions

• RQ1 Can gradient-based unlearning effectively reduce the memorization of sensitive sequences in large language models without degrading general performance?
• RQ2 How does sequential unlearning compare to unlearning all data at once in terms of effectiveness and efficiency?
• RQ3 To what extent does the domain of the data being unlearned affect the difficulty and success of knowledge unlearning?
• RQ4 How does knowledge unlearning compare to existing privacy mitigation methods like data deduplication and differential privacy decoding in terms of privacy guarantees and computational cost?
• RQ5 What metrics and thresholds can empirically define when a memorized sequence has been successfully forgotten?

Key findings

• Knowledge unlearning reduced extraction likelihood (EL) below the forgetting threshold (e.g., 5.68% for EL₁₀ on GPT-Neo 1.3B) after unlearning, indicating effective forgetting.
• The method caused little to no degradation in general language modeling performance, with average accuracy across 9 benchmarks remaining stable or improving slightly.
• Sequential unlearning outperformed single-shot unlearning, with later chunks forgotten in just one or two epochs due to sequence similarity.
• Performance on 9 classification tasks remained high (e.g., 49.93% average accuracy for EL₅ on 1.3B model), showing robustness to unlearning.
• The approach provided stronger empirical privacy guarantees than data deduplication and differential privacy decoding, while being significantly more efficient.
• Unlearning success varied by data domain, indicating that memorization and forgetting are highly dependent on the semantic and structural characteristics of the target data.