[Paper Review] What Does it Mean for a Language Model to Preserve Privacy?

Hannah Brown, Katherine Lee+3 more|arXiv (Cornell University)|Feb 11, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper argues that current privacy-preserving techniques for language models—such as data sanitization and differential privacy—fail to meet human expectations of privacy due to their narrow assumptions about data structure and sensitivity. The authors conclude that only language models trained on text explicitly intended for public use can meaningfully preserve privacy, as contextual integrity and informed consent are unattainable with existing methods.

ABSTRACT

Natural language reflects our private lives and identities, making its privacy concerns as broad as those of real life. Language models lack the ability to understand the context and sensitivity of text, and tend to memorize phrases present in their training sets. An adversary can exploit this tendency to extract training data. Depending on the nature of the content and the context in which this data was collected, this could violate expectations of privacy. Thus there is a growing interest in techniques for training language models that preserve privacy. In this paper, we discuss the mismatch between the narrow assumptions made by popular data protection techniques (data sanitization and differential privacy), and the broadness of natural language and of privacy as a social norm. We argue that existing protection methods cannot guarantee a generic and meaningful notion of privacy for language models. We conclude that language models should be trained on text data which was explicitly produced for public use.

Motivation & Objective

• To analyze the mismatch between technical privacy protections and human expectations of privacy in natural language.
• To critique the limitations of data sanitization and differential privacy in protecting sensitive language data.
• To argue that informed consent is practically unattainable for users due to the collective and contextual nature of private information in text.
• To propose that language models should be trained exclusively on data explicitly intended for public use.
• To advocate for a shift from data protection to privacy-preserving training on public-intended text.

Proposed method

• The paper conducts a conceptual and normative analysis of privacy in language models, contrasting technical assumptions with social norms of privacy.
• It evaluates data sanitization by examining its inability to recognize semantically sensitive content beyond formal patterns.
• It critiques differential privacy by highlighting its reliance on well-defined data records, which do not align with the fluid, group-based nature of private language.
• It introduces the concept of contextual integrity as a benchmark for privacy, requiring models to respect the original context of data use.
• It proposes a practical alternative: training language models only on text explicitly intended for public dissemination.
• It suggests local fine-tuning on private data as a complementary approach to preserve user privacy in personalized settings.

Experimental results

Research questions

• RQ1 Can data sanitization techniques reliably identify and remove all forms of private information from natural language text?
• RQ2 To what extent does differential privacy provide meaningful privacy guarantees for natural language data with ambiguous or collective boundaries?
• RQ3 Can users meaningfully give informed consent for their private text to be used in language model training?
• RQ4 What constitutes a 'meaningful' privacy guarantee in the context of language models, given human social norms?
• RQ5 Is it possible to achieve true privacy preservation in language models without relying on data protection techniques alone?

Key findings

• Data sanitization fails to recognize most semantically private information, as it relies on formal specifications that do not capture contextual sensitivity.
• Differential privacy provides only limited protection because it assumes well-defined, individual data records, which do not reflect the collective and contextual nature of language.
• Users cannot give informed consent for their data to be used in language models, as they are often unaware of how their data is stored, processed, or shared.
• Sensitive information in text is frequently shared among multiple individuals, making individual consent insufficient and impractical.
• Language models trained on publicly accessible data may still violate privacy, as such data is not necessarily intended for public use.
• The only viable path to meaningful privacy preservation is training language models exclusively on text explicitly intended for public dissemination.