[Paper Review] Whispers in the Machine: Confidentiality in Agentic Systems

Jonathan Evertz, Merlin Chlosta+2 more|arXiv (Cornell University)|Feb 10, 2024

Digital Rights Management and Security4 citations

TL;DR

This paper introduces a formal 'secret key' game to evaluate and improve confidentiality in LLM-integrated systems, where LLMs interact with external tools. It proposes robustness fine-tuning via black-box prompt-based adversarial training, reducing attack success rates by up to 50% on individual attacks and improving resilience against unknown threats, with a 14%pt. average reduction when combined with input/output sanitization.

ABSTRACT

The interaction between users and applications is increasingly shifted toward natural language by deploying Large Language Models (LLMs) as the core interface. The capabilities of these so-called agents become more capable the more tools and services they serve as an interface for, ultimately leading to agentic systems. Agentic systems use LLM-based agents as interfaces for most user interactions and various integrations with external tools and services. While these interfaces can significantly enhance the capabilities of the agentic system, they also introduce a new attack surface. Manipulated integrations, for example, can exploit the internal LLM and compromise sensitive data accessed through other interfaces. While previous work primarily focused on attacks targeting a model's alignment or the leakage of training data, the security of data that is only available during inference has escaped scrutiny so far. In this work, we demonstrate how the integration of LLMs into systems with external tool integration poses a risk similar to established prompt-based attacks, able to compromise the confidentiality of the entire system. Introducing a systematic approach to evaluate these confidentiality risks, we identify two specific attack scenarios unique to these agentic systems and formalize these into a tool-robustness framework designed to measure a model's ability to protect sensitive information. Our analysis reveals significant vulnerabilities across all tested models, highlighting an increased risk when models are combined with external tools.

Motivation & Objective

• To address the lack of formal evaluation for confidentiality in LLM-integrated systems, where private data may leak across tool interfaces.
• To systematically assess the effectiveness of existing prompt-based attacks and defenses on state-of-the-art LLMs.
• To develop a generalizable defense mechanism that enhances LLM resilience against both known and unknown confidentiality attacks.
• To demonstrate that robustness fine-tuning, inspired by adversarial training, can significantly reduce the success rate of secret-extraction attacks.

Proposed method

• Formalize a 'secret key' game where a model must conceal a randomly generated secret string during interaction, simulating real-world data confidentiality.
• Implement eight prompt-based attack strategies to extract the secret, including jailbreaking, role-playing, and instruction injection techniques.
• Evaluate four existing defense mechanisms: input sanitization, output filtering, prompt paraphrasing, and detection via perplexity thresholds.
• Propose robustness fine-tuning using black-box adversarial prompts—generated via attacks—to fine-tune the LLM on malicious inputs without requiring white-box access.
• Apply Parameter-Efficient Fine-Tuning (PEFT) via QLoRA to reduce computational cost while maintaining performance.
• Combine fine-tuned models with input/output sanitization for layered defense, measuring cumulative effectiveness.

Experimental results

Research questions

• RQ1 How effective are existing prompt-based attacks in extracting a secret key from LLMs in integrated systems?
• RQ2 To what extent do current defense mechanisms generalize across diverse attack strategies?
• RQ3 Can robustness fine-tuning improve LLM resilience against both known and unknown confidentiality attacks?
• RQ4 What is the impact of combining fine-tuning with input/output sanitization on overall attack success rate?

Key findings

• Without defenses, the most effective attacks achieved a 61% success rate in extracting the secret key from state-of-the-art LLMs.
• Current defenses mitigate some attacks but fail to generalize, with many remaining effective across different attack strategies.
• Robustness fine-tuning against a single attack reduced the average attack success rate from 26.5% to 12.75%, a 13.75 percentage point reduction.
• When fine-tuned against all eight attacks simultaneously, the success rate dropped by 9 percentage points on average.
• Cross-validation showed increased robustness against previously unseen attacks, indicating improved generalization.
• Combining fine-tuning with input/output sanitization reduced the average attack success rate by 14 percentage points.