[Paper Review] Knowledge Sanitization of Large Language Models

Yoïchi Ishibashi, Hidetoshi Shimodaira|arXiv (Cornell University)|Sep 21, 2023

Topic Modeling4 citations

TL;DR

This paper proposes knowledge sanitization, a fine-tuning method that redirects large language models (LLMs) to generate predefined harmless responses—like "I don’t know"—when queried about sensitive or confidential information. The approach effectively blocks knowledge leakage and extraction attacks while preserving the model’s overall performance on unrelated knowledge tasks.

ABSTRACT

We explore a knowledge sanitization approach to mitigate the privacy concerns associated with large language models (LLMs). LLMs trained on a large corpus of Web data can memorize and potentially reveal sensitive or confidential information, raising critical security concerns. Our technique efficiently fine-tunes these models using the Low-Rank Adaptation (LoRA) method, prompting them to generate harmless responses such as ``I don't know'' when queried about specific information. Experimental results in a closed-book question-answering task show that our straightforward method not only minimizes particular knowledge leakage but also preserves the overall performance of LLMs. These two advantages strengthen the defense against extraction attacks and reduces the emission of harmful content such as hallucinations.

Motivation & Objective

• To address privacy risks in LLMs caused by memorization of personal and confidential information from web training data.
• To prevent adversarial extraction attacks that exploit LLMs to retrieve sensitive data such as names, addresses, and email addresses.
• To develop a method that not only blocks harmful outputs but also ensures generation of safe, predefined responses to reduce secondary risks like hallucinations.
• To maintain the LLM’s general knowledge capabilities and performance on non-sensitive queries after sanitization.
• To provide a post-training, fine-tuning-based solution that does not require retraining or architectural changes.

Proposed method

• The method applies instruction tuning to fine-tune pre-trained LLMs using adversarial examples that trigger sensitive information queries.
• It trains the model to output a fixed harmless token sequence—such as "I don’t know"—in response to prompts targeting specific confidential knowledge.
• The approach uses a prompt template to generate adversarial examples, including context-rich sentences ending with "is ___", to test and train the model’s resistance to information extraction.
• The fine-tuning process is applied directly to existing LLMs, such as LLaMA and GPT-J, without retraining from scratch.
• The model is evaluated on closed-book question-answering tasks to assess both sanitization effectiveness and retention of general knowledge.
• The method is designed to be robust against prompt engineering attacks, including jailbreaks and suffix attacks, by consistently rejecting sensitive queries with safe responses.

Experimental results

Research questions

• RQ1 Can knowledge sanitization effectively prevent LLMs from leaking specific confidential information such as personal names, addresses, and email addresses?
• RQ2 Does the method preserve the LLM’s ability to answer factual questions outside the sanitized knowledge domains?
• RQ3 How robust is the sanitized model against adversarial extraction attacks, including prompt-based and context-rich attacks?
• RQ4 Can the approach be applied to existing LLMs without retraining, using only fine-tuning?
• RQ5 Does the use of predefined harmless responses reduce the risk of hallucinations or harmful content generation?

Key findings

• The sanitized LLM consistently responded with "I don’t know" or blank responses when queried about sensitive information, such as Julius Caesar’s name or wife, effectively blocking data leakage.
• In extraction attack experiments, the sanitized model refrained from generating any information related to Julius Caesar, even when prompted with contextually rich sentences, demonstrating strong resistance to adversarial queries.
• The model maintained high performance on unrelated knowledge tasks, such as answering questions about Cleopatra and Pompeii, confirming that general knowledge capabilities were preserved.
• The method showed robustness against various prompt engineering techniques, including jailbreaks and suffix attacks, by consistently generating safe responses instead of sensitive content.
• The approach successfully mitigated information leakage without introducing hallucinations, as the model did not generate substitute or fabricated information.
• The fine-tuning process was effective and efficient, enabling knowledge sanitization without retraining the base model, making it applicable to existing LLMs.