[Paper Review] Anonymity at Risk? Assessing Re-Identification Capabilities of Large Language Models

Alex Nyffenegger, Matthias Stürmer+1 more|arXiv (Cornell University)|Aug 22, 2023

Hate Speech and Cyberbullying Detection4 citations

TL;DR

This study evaluates the re-identification risk of large language models (LLMs) in anonymized Swiss court rulings and Wikipedia articles, using a proof-of-concept framework with state-of-the-art models. Despite high success on Wikipedia, LLMs struggle with real court decisions due to data sparsity and complexity, though model size, input length, and instruction tuning significantly influence performance, suggesting future risks that necessitate improved anonymization safeguards.

ABSTRACT

Anonymity of both natural and legal persons in court rulings is a critical aspect of privacy protection in the European Union and Switzerland. With the advent of LLMs, concerns about large-scale re-identification of anonymized persons are growing. In accordance with the Federal Supreme Court of Switzerland, we explore the potential of LLMs to re-identify individuals in court rulings by constructing a proof-of-concept using actual legal data from the Swiss federal supreme court. Following the initial experiment, we constructed an anonymized Wikipedia dataset as a more rigorous testing ground to further investigate the findings. With the introduction and application of the new task of re-identifying people in texts, we also introduce new metrics to measure performance. We systematically analyze the factors that influence successful re-identifications, identifying model size, input length, and instruction tuning among the most critical determinants. Despite high re-identification rates on Wikipedia, even the best LLMs struggled with court decisions. The complexity is attributed to the lack of test datasets, the necessity for substantial training resources, and data sparsity in the information used for re-identification. In conclusion, this study demonstrates that re-identification using LLMs may not be feasible for now, but as the proof-of-concept on Wikipedia showed, it might become possible in the future. We hope that our system can help enhance the confidence in the security of anonymized decisions, thus leading to the courts being more confident to publish decisions.

Motivation & Objective

• To evaluate the feasibility of re-identifying anonymized individuals in Swiss federal court decisions using large language models.
• To investigate the factors influencing LLM performance in re-identification tasks, including model size, input length, and instruction tuning.
• To develop a responsible, court-collaborative tool to test the robustness of current anonymization practices against emerging LLM capabilities.
• To improve anonymization recall from 83% to 93% by fine-tuning a legal-specific BERT model in collaboration with the Swiss Federal Supreme Court.
• To inform legal institutions, privacy advocates, and NLP practitioners about evolving privacy risks in published legal documents under GDPR and similar regulations.

Proposed method

• Constructed a proof-of-concept re-identification system using LLMs on anonymized Swiss federal court rulings and a curated Wikipedia dataset for controlled testing.
• Applied a new task formulation for re-identification, introducing metrics such as PNMS (Precision of Name Matching Score) and NLD (Normalized Levenshtein Distance) to measure performance.
• Evaluated 25+ LLMs across varying sizes (from 0.08B to 1800B parameters), including GPT-4, LLaMA-2, BLOOM, and legal-specific models like Legal-Swiss-RobBERTa.
• Used beam search and top-5 prediction strategies on the first 1,000 characters of input text to assess re-identification accuracy.
• Pre-trained a legal-specific model on Swiss legal text (Swissdox) to improve anonymization recall, increasing it from 83% to 93%.
• Conducted ablation studies to isolate the impact of model size, input length, and instruction tuning on re-identification success.

Figure 1: Re-identification rate by parameter count

Experimental results

Research questions

• RQ1 RQ1: How effectively can various LLMs re-identify masked persons within Wikipedia pages and in Swiss court rulings?
• RQ2 RQ2: What are the key factors that influence the performance of LLMs in re-identification tasks?
• RQ3 RQ3: How will evolving LLM capabilities and their use in re-identifications affect the preservation of privacy in anonymized court rulings in Switzerland?

Key findings

• On the Wikipedia dataset, the best-performing LLMs (e.g., GPT-4) achieved a PNMS of 0.71, indicating strong re-identification capability on curated, information-rich texts.
• In real court rulings, even the most capable models (e.g., GPT-4) achieved only a PNMS of 0.71, but with significantly higher NLD (0.17), indicating poor alignment with ground truth names.
• Model size, input length, and instruction tuning were identified as the most critical factors influencing re-identification success, with larger and instruction-tuned models performing significantly better.
• The best-performing model (GPT-4) achieved a W-PNMS of 0.65 on Wikipedia, but only 0.14 on court rulings, highlighting the gap in performance due to data sparsity and complexity.
• Only when provided with a highly curated set of relevant news articles could models successfully re-identify anonymized defendants, underscoring the role of external context in re-identification.
• The study demonstrates that while current LLMs are not yet effective at re-identifying individuals in anonymized court decisions, the risk is non-zero and may grow with model advancement and better data integration.

Figure 2: Categorized predictions for rulings