[Paper Review] Are Chatbots Ready for Privacy-Sensitive Applications? An Investigation into Input Regurgitation and Prompt-Induced Sanitization

Aman Priyanshu, Supriti Vijay+3 more|arXiv (Cornell University)|May 24, 2023

Privacy, Security, and Data Protection11 citations

TL;DR

The paper shows ChatGPT can memorize and regurgitate PII/PHI, but prompt-induced sanitization can substantially reduce leakage and improve compliance with HIPAA and GDPR, with biases across subgroups and open datasets for further research.

ABSTRACT

LLM-powered chatbots are becoming widely adopted in applications such as healthcare, personal assistants, industry hiring decisions, etc. In many of these cases, chatbots are fed sensitive, personal information in their prompts, as samples for in-context learning, retrieved records from a database, or as part of the conversation. The information provided in the prompt could directly appear in the output, which might have privacy ramifications if there is sensitive information there. As such, in this paper, we aim to understand the input copying and regurgitation capabilities of these models during inference and how they can be directly instructed to limit this copying by complying with regulations such as HIPAA and GDPR, based on their internal knowledge of them. More specifically, we find that when ChatGPT is prompted to summarize cover letters of a 100 candidates, it would retain personally identifiable information (PII) verbatim in 57.4% of cases, and we find this retention to be non-uniform between different subgroups of people, based on attributes such as gender identity. We then probe ChatGPT's perception of privacy-related policies and privatization mechanisms by directly instructing it to provide compliant outputs and observe a significant omission of PII from output.

Motivation & Objective

• Assess whether ChatGPT copies Personal Identifiable Information (PII) from prompts and prior interactions (input regurgitation).
• Evaluate how direct instruction to comply with privacy regulations affects outputs (prompt-induced sanitization).
• Quantify leakage and utility trade-offs across two domain case studies (hiring with PII and healthcare with PHI).
• Analyze subgroup variations (e.g., gender identity) in leakage and sanitization efficacy.
• Open-source datasets of PII/PHI-infused samples to enable further research.

Proposed method

• Two case studies: hiring decisions with PII in cover letters and healthcare assistance with PHI in medical notes.
• Prompt prompting to induce HIPAA and GDPR compliance, including k-anonymity-based sanitization prompts.
• Synthetic datasets: HIPAA/PHI-infused medical notes (via MIMIC-III augmentation) and PII-infused cover letters.
• Quantitative evaluation using privacy leakage and utility metrics (Boolean privacy leakage, Jaro distance, BLEU, age matching).
• Analysis of attribute correlations (gender, date of birth, university) and their impact on leakage.

Figure 1: Our experimental setup and methodology, where we first quantify ChatGPT’s capability to copy and retain personally identifiable information (left). Then, we instruct ChatGPT to sanitize its output using k-anonymity, and to abide by privacy policies (HIPAA).

Experimental results

Research questions

• RQ1 To what extent does the model copy PII/PHI from prompts or prior conversations?
• RQ2 Can explicit privacy-compliance prompts reduce PII/PHI leakage, and to what extent?
• RQ3 How does leakage vary across sensitive attributes (e.g., gender, DoB, university) and across domains (medical vs. hiring)?
• RQ4 What is the impact of sanitization on the utility of the outputs for downstream tasks?

Key findings

• ChatGPT copies PII 57.4% of the time baseline; this drops to 30.5% with compliance prompts and to 15.2% with compliance prompts plus explicit scrub instructions.
• Prompt-induced sanitization reduces PHI leakage to 26.4% of baseline (a 56% reduction) in the medical dataset.
• In the hiring dataset, prompt-based sanitization yields about a 30.2% reduction in PII leakage relative to baseline.
• Leakage and sanitization outcomes are non-uniform across subgroups, with notably less PII copying for non-binary individuals.
• Prompts generate anonymized outputs (e.g., replacing names with identifiers like Patient-1 S. or Unknown) and, in some cases, complete redaction.
• Utility analyses show retained non-sensitive attributes and varying retention for sensitive attributes across prompts.

Figure 2: This image showcases the utility analysis of skills and hireability of role in the hiring dataset (left) & of symptoms and diagnosis for the medical dataset (right).