[Paper Review] PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action

Yijia Shao, Tianshi Li+3 more|arXiv (Cornell University)|Aug 29, 2024

Privacy, Security, and Data Protection4 citations

TL;DR

PrivacyLens proposes a novel framework to evaluate language models' privacy norm awareness in real-world agent applications by generating expressive vignettes and agent trajectories from privacy-sensitive seeds. It reveals that even state-of-the-art models like GPT-4 and Llama-3-70B leak sensitive information in 25.68% and 38.69% of cases, respectively, despite privacy-enhancing prompts, highlighting a critical gap between probing performance and actual behavior.

ABSTRACT

As language models (LMs) are widely utilized in personalized communication scenarios (e.g., sending emails, writing social media posts) and endowed with a certain level of agency, ensuring they act in accordance with the contextual privacy norms becomes increasingly critical. However, quantifying the privacy norm awareness of LMs and the emerging privacy risk in LM-mediated communication is challenging due to (1) the contextual and long-tailed nature of privacy-sensitive cases, and (2) the lack of evaluation approaches that capture realistic application scenarios. To address these challenges, we propose PrivacyLens, a novel framework designed to extend privacy-sensitive seeds into expressive vignettes and further into agent trajectories, enabling multi-level evaluation of privacy leakage in LM agents' actions. We instantiate PrivacyLens with a collection of privacy norms grounded in privacy literature and crowdsourced seeds. Using this dataset, we reveal a discrepancy between LM performance in answering probing questions and their actual behavior when executing user instructions in an agent setup. State-of-the-art LMs, like GPT-4 and Llama-3-70B, leak sensitive information in 25.68% and 38.69% of cases, even when prompted with privacy-enhancing instructions. We also demonstrate the dynamic nature of PrivacyLens by extending each seed into multiple trajectories to red-team LM privacy leakage risk. Dataset and code are available at https://github.com/SALT-NLP/PrivacyLens.

Motivation & Objective

• To address the lack of realistic evaluation frameworks for measuring privacy norm awareness in language model agents.
• To identify the disconnect between LMs’ performance on probing questions and their actual behavior in agent workflows.
• To develop a scalable, extensible framework that captures context-dependent privacy norms in tool-augmented LM interactions.
• To quantify the risk of unintentional privacy leakage in LM-mediated communication, especially when models access sensitive data via tools like calendars or emails.

Proposed method

• Ground privacy-sensitive seeds in privacy literature and crowdsourced inputs using the Contextual Integrity theory to define norms across data subject, sender, recipient, and transmission method.
• Expand each seed into expressive, context-rich vignettes using template-based generation to simulate realistic agent scenarios.
• Simulate agent trajectories by orchestrating tool-use actions (e.g., calendar reading, email sending) based on user instructions and observed data.
• Use a multi-level evaluation pipeline to assess whether the final agent actions violate established privacy norms.
• Red-team the models by generating multiple diverse trajectories per seed to uncover worst-case leakage scenarios.
• Maintain an open-source, extensible framework with dataset and code available on GitHub for community use and extension.

Experimental results

Research questions

• RQ1 To what extent do language models violate privacy norms when acting as agents in real communication workflows, even when prompted with privacy-preserving instructions?
• RQ2 How does the performance of LMs on privacy probing questions correlate with their actual behavior in agent trajectories?
• RQ3 What is the prevalence of unintentional privacy leakage in LM-mediated communication across diverse, context-sensitive scenarios?
• RQ4 How can privacy norm awareness be systematically evaluated in agentic workflows involving tool use and sensitive data access?
• RQ5 Can the framework be extended to red-team models by generating adversarial trajectories that expose privacy leakage risks?

Key findings

• State-of-the-art language models such as GPT-4 and Llama-3-70B leak sensitive information in 25.68% and 38.69% of agent trajectories, respectively, despite privacy-enhancing prompts.
• A significant discrepancy exists between LMs’ performance on privacy probing questions and their actual behavior in agent workflows, indicating that probing tasks alone are insufficient for assessing privacy safety.
• The framework successfully generates diverse, contextually rich agent trajectories that expose rare but consequential privacy leakage cases.
• The PrivacyLens framework enables dynamic red-teaming of LMs by generating multiple trajectories per seed, uncovering hidden privacy risks in tool-augmented agent behavior.
• The dataset and code are publicly available under a CC BY license, enabling community use and extension for evaluating privacy norm awareness in agentic LMs.