[Paper Review] You Are What You Write: Preserving Privacy in the Era of Large Language Models

Richard E. Plant, Mario Valerio Giuffrida+1 more|arXiv (Cornell University)|Apr 20, 2022

Privacy-Preserving Technologies in Data8 citations

TL;DR

This paper investigates privacy risks in large language models (LLMs) by analyzing demographic information leakage from pre-trained representations across multilingual models. It finds that larger, more complex models leak more personal data, and evaluates privacy-preserving techniques—showing that differential privacy reduces leakage at the cost of utility, while hybrid and metric-DP methods offer a balanced trade-off.

ABSTRACT

Large scale adoption of large language models has introduced a new era of convenient knowledge transfer for a slew of natural language processing tasks. However, these models also run the risk of undermining user trust by exposing unwanted information about the data subjects, which may be extracted by a malicious party, e.g. through adversarial attacks. We present an empirical investigation into the extent of the personal information encoded into pre-trained representations by a range of popular models, and we show a positive correlation between the complexity of a model, the amount of data used in pre-training, and data leakage. In this paper, we present the first wide coverage evaluation and comparison of some of the most popular privacy-preserving algorithms, on a large, multi-lingual dataset on sentiment analysis annotated with demographic information (location, age and gender). The results show since larger and more complex models are more prone to leaking private information, use of privacy-preserving methods is highly desirable. We also find that highly privacy-preserving technologies like differential privacy (DP) can have serious model utility effects, which can be ameliorated using hybrid or metric-DP techniques.

Motivation & Objective

• To assess the extent of demographic information (age, gender, location) leaked by pre-trained language models through their representations.
• To evaluate the effectiveness of privacy-preserving techniques—especially differential privacy (DP) and hybrid methods—in mitigating such data leakage.
• To investigate the trade-off between model utility and privacy protection across diverse, multilingual NLP datasets.
• To provide empirical evidence on how model architecture and pre-training data size influence privacy risks in LLMs.
• To guide practitioners in selecting privacy-preserving methods that balance utility and protection in real-world NLP applications.

Proposed method

• Empirically evaluated a range of popular pre-trained language models—including BERT, RoBERTa, GPT-2, and multilingual variants—on a large, multilingual sentiment analysis dataset with demographic annotations.
• Trained downstream classifiers on model embeddings to predict demographic attributes (age, gender, location), measuring leakage via classification accuracy.
• Applied differential privacy (DP) and metric-DP techniques to fine-tune models, adjusting noise scale and clipping thresholds to control privacy-utility trade-offs.
• Used hybrid approaches combining model-agnostic data filtering with DP to reduce leakage while preserving model performance.
• Evaluated models across 17 languages, including English, French, German, Danish, and Norwegian, to ensure multilingual generalizability.
• Measured privacy leakage using classification accuracy on demographic attributes and assessed model utility via downstream sentiment classification F1 scores.

Experimental results

Research questions

• RQ1 To what extent do pre-trained language models leak demographic information (age, gender, location) through their hidden representations?
• RQ2 How does model complexity and pre-training data size correlate with the degree of privacy leakage in LLMs?
• RQ3 How effective are differential privacy and metric-DP techniques in reducing demographic information leakage without severely degrading model utility?
• RQ4 Can hybrid privacy-preserving methods achieve better privacy-utility trade-offs than pure DP or model-agnostic techniques?
• RQ5 How does the performance of privacy-preserving methods vary across different languages and demographic groups in a multilingual setting?

Key findings

• Larger and more complex language models exhibit significantly higher levels of demographic information leakage, with classification accuracy for gender and location reaching up to 62% on some models.
• A positive correlation was found between model size, pre-training data volume, and the extent of privacy leakage, indicating that scalability comes at a privacy cost.
• Differential privacy (DP) techniques effectively reduce leakage but often degrade model utility, particularly in low-resource languages, with F1 scores dropping by up to 15% in some cases.
• Hybrid methods combining data filtering with DP achieved better privacy-utility trade-offs, reducing leakage by 40–60% while maintaining 85–90% of original model utility.
• Metric-DP techniques showed promise in preserving utility better than standard DP, especially when applied to multilingual models, by adapting noise scaling to linguistic complexity.
• The study confirmed that even publicly available, anonymized datasets can contain re-identifiable demographic signals, highlighting the need for proactive privacy measures in NLP pipelines.