[Paper Review] The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Risks

Xiaoyi Chen, Siyuan Tang+8 more|arXiv (Cornell University)|Oct 24, 2023

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper introduces the Janus attack, a novel method that exploits fine-tuning in large language models (LLMs) to recover and disclose previously hidden Personal Identifiable Information (PII), such as email addresses. By fine-tuning GPT-3.5 on just 10 PII examples, the attack achieves a 69.9% precision in recovering concealed emails, demonstrating that fine-tuning can reverse catastrophic forgetting and expose private data despite alignment safeguards.

ABSTRACT

The rapid advancements of large language models (LLMs) have raised public concerns about the privacy leakage of personally identifiable information (PII) within their extensive training datasets. Recent studies have demonstrated that an adversary could extract highly sensitive privacy data from the training data of LLMs with carefully designed prompts. However, these attacks suffer from the model's tendency to hallucinate and catastrophic forgetting (CF) in the pre-training stage, rendering the veracity of divulged PIIs negligible. In our research, we propose a novel attack, Janus, which exploits the fine-tuning interface to recover forgotten PIIs from the pre-training data in LLMs. We formalize the privacy leakage problem in LLMs and explain why forgotten PIIs can be recovered through empirical analysis on open-source language models. Based upon these insights, we evaluate the performance of Janus on both open-source language models and two latest LLMs, i.e., GPT-3.5-Turbo and LLaMA-2-7b. Our experiment results show that Janus amplifies the privacy risks by over 10 times in comparison with the baseline and significantly outperforms the state-of-the-art privacy extraction attacks including prefix attacks and in-context learning (ICL). Furthermore, our analysis validates that existing fine-tuning APIs provided by OpenAI and Azure AI Studio are susceptible to our Janus attack, allowing an adversary to conduct such an attack at a low cost.

Motivation & Objective

• To investigate whether fine-tuning LLMs can trigger the recovery and disclosure of PII that was previously forgotten due to catastrophic forgetting.
• To evaluate the effectiveness of fine-tuning as a means to bypass alignment-based privacy protections such as RLHF.
• To explore the feasibility of reconstructing concealed PII using only a small number of PII instances for fine-tuning.
• To assess the risk of privacy leakage through the fine-tuning interface in production LLMs like GPT-3.5.
• To propose a new attack vector—Janus—that leverages fine-tuning to recover forgotten PII, highlighting a critical privacy threat in LLM deployment.

Proposed method

• The Janus attack constructs a PII association task where the model is fine-tuned on a minimal dataset of PII instances (e.g., 10 email-address pairs).
• The attack uses prompts that map a person's name to their email, training the LLM to associate names with specific PII.
• Fine-tuning is applied after the model has undergone catastrophic forgetting, effectively recovering the PII association task.
• The method evaluates performance using precision and hit rate metrics on both Enron and non-Enron email datasets.
• The approach compares against jailbreaking techniques to isolate the unique effectiveness of fine-tuning in reversing forgetting.
• Experiments use GPT-3.5-Turbo and GPT-2 models, with continual learning simulating real-world multi-task scenarios.

Experimental results

Research questions

• RQ1 Can fine-tuning on a small number of PII instances recover and expose previously forgotten personal information in LLMs?
• RQ2 To what extent does the Janus attack outperform traditional jailbreaking methods in recovering PII?
• RQ3 Does the effectiveness of the Janus attack vary across different PII types or domains, such as Enron vs. non-Enron emails?
• RQ4 How does catastrophic forgetting impact the model’s ability to retain PII, and can fine-tuning reverse this loss?
• RQ5 Can alignment mechanisms like RLHF be circumvented through fine-tuning to enable PII disclosure?

Key findings

• Fine-tuning GPT-3.5 on just 10 PII examples achieved a 69.9% precision in recovering concealed email addresses from the Enron dataset.
• The Janus attack achieved a 100% bypass rate of RLHF alignment safeguards, demonstrating full success in circumventing privacy protections.
• In contrast, jailbreaking methods achieved only 6% hit rate and 0% precision on the same task, indicating poor performance in PII recovery despite bypassing alignment.
• On the more complex non-Enron dataset, the Janus attack achieved a precision of 1.9% with 100 examples, indicating domain-specific challenges in name and domain extraction.
• The attack remained effective even with minimal fine-tuning data, showing that recovery of forgotten PII is feasible with low-cost interventions.
• The results confirm that catastrophic forgetting, while harmful to model performance, can be exploited via fine-tuning to restore and expose sensitive PII.