[Paper Review] Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks

Vaidehi Patil, Peter Hase+1 more|arXiv (Cornell University)|Sep 29, 2023

Topic Modeling4 citations

TL;DR

This paper investigates whether sensitive information can be truly deleted from large language models (LLMs) by directly editing model weights, proposing a novel attack-and-defense framework. Despite state-of-the-art editing methods like ROME and MEMIT, attacks recover deleted facts in 38–89% of cases, revealing that residual traces in hidden states and question rephrasing undermine deletion efficacy, highlighting the difficulty of achieving robust, privacy-preserving model editing.

ABSTRACT

Pretrained language models sometimes possess knowledge that we do not wish them to, including memorized personal information and knowledge that could be used to harm people. They can also output toxic or harmful text. To mitigate these safety and informational issues, we propose an attack-and-defense framework for studying the task of deleting sensitive information directly from model weights. We study direct edits to model weights because (1) this approach should guarantee that particular deleted information is never extracted by future prompt attacks, and (2) it should protect against whitebox attacks, which is necessary for making claims about safety/privacy in a setting where publicly available model weights could be used to elicit sensitive information. Our threat model assumes that an attack succeeds if the answer to a sensitive question is located among a set of B generated candidates, based on scenarios where the information would be insecure if the answer is among B candidates. Experimentally, we show that even state-of-the-art model editing methods such as ROME struggle to truly delete factual information from models like GPT-J, as our whitebox and blackbox attacks can recover "deleted" information from an edited model 38% of the time. These attacks leverage two key observations: (1) that traces of deleted information can be found in intermediate model hidden states, and (2) that applying an editing method for one question may not delete information across rephrased versions of the question. Finally, we provide new defense methods that protect against some extraction attacks, but we do not find a single universally effective defense method. Our results suggest that truly deleting sensitive information is a tractable but difficult problem, since even relatively low attack success rates have potentially severe societal implications for real-world deployment of language models.

Motivation & Objective

• To address the critical need for reliable deletion of sensitive information from LLMs, including personal data and harmful knowledge, to support privacy rights and safety.
• To develop a rigorous threat model where an attack succeeds if a deleted fact is recoverable among a small set of B candidates, reflecting real-world adversarial scenarios.
• To evaluate whether direct model editing—altering weights to delete information—is a viable and secure alternative to RLHF or data-level interventions.
• To test the robustness of proposed defenses against both intended and unforeseen extraction attacks, especially whitebox probing.
• To demonstrate that even advanced editing methods fail to fully erase information, challenging assumptions about model privacy and safety.

Proposed method

• Proposes a dual attack-and-defense framework to test the effectiveness of model editing for deleting sensitive factual information from LLM weights.
• Designs two whitebox attacks: the Head Projection attack and the Probability Delta attack, which exploit residual information in model hidden states and attention heads.
• Introduces a blackbox attack that queries the model with rephrased versions of deleted questions to test generalization of deletion across semantic variants.
• Employs multiple defense strategies: Fact Erasure, Empty Response Injection, Error Injection, Head Projection, Max-Entropy regularization, and IR-based defenses to harden model weights.
• Uses GPT-J as the target model and evaluates on CounterFact and zsRE datasets, measuring attack success via recovery rates among B candidates.
• Applies a threat model where success is defined as recovering the correct answer from a small set B of generated responses, simulating plausible adversarial settings.

Experimental results

Research questions

• RQ1 Can sensitive factual information be reliably deleted from LLM weights using current model editing techniques?
• RQ2 To what extent do whitebox and blackbox extraction attacks succeed in recovering information from models that were supposedly edited to delete it?
• RQ3 How effective are proposed defense mechanisms in protecting against both intended and unforeseen extraction attacks?
• RQ4 Does the presence of residual information in hidden states or attention heads undermine the effectiveness of model editing for deletion?
• RQ5 What are the implications of partial deletion for privacy, safety, and compliance with data ownership rights in LLM deployment?

Key findings

• Even state-of-the-art editing methods like ROME and MEMIT fail to fully delete factual information, with attacks recovering deleted facts in up to 89.65% of cases under the Probability Delta attack.
• The Head Projection attack achieves a success rate of 84.80% against the IR defense and 56.83% against the Head Projection defense, showing that defenses are not universally robust.
• The Max-Entropy defense reduces whitebox attack success to 2.42% (ROME) and 38.11% (MEMIT), but blackbox attacks still succeed at 28.00% (CounterFact) and 19.42% (zsRE), indicating persistent vulnerability.
• The Probability Delta attack, not designed to target the Max-Entropy defense, still achieves 2.42% success against it, showing that some defenses offer incidental protection against unforeseen attacks.
• The Empty Response Injection defense achieves 99.78% success in blocking attacks on zsRE with ROME, but fails to generalize, as seen in 88.55% success for the same attack on MEMIT.
• No single defense method is effective against all attacks, and even the best defenses leave significant recovery rates, suggesting that true deletion remains a hard problem.