[Paper Review] Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems

Dong-Hyun Lee, Mansi Tiwari|arXiv (Cornell University)|Oct 9, 2024

Simulation Techniques and Applications4 citations

TL;DR

This paper introduces Prompt Infection, a novel LLM-to-LLM prompt injection attack that enables self-replicating malicious prompts to spread across interconnected agents in multi-agent systems, leading to data theft and system compromise. The authors demonstrate that combining LLM Tagging with existing defense mechanisms significantly reduces infection success rates, highlighting the need for integrated security strategies in multi-agent LLM architectures.

ABSTRACT

As Large Language Models (LLMs) grow increasingly powerful, multi-agent systems are becoming more prevalent in modern AI applications. Most safety research, however, has focused on vulnerabilities in single-agent LLMs. These include prompt injection attacks, where malicious prompts embedded in external content trick the LLM into executing unintended or harmful actions, compromising the victim's application. In this paper, we reveal a more dangerous vector: LLM-to-LLM prompt injection within multi-agent systems. We introduce Prompt Infection, a novel attack where malicious prompts self-replicate across interconnected agents, behaving much like a computer virus. This attack poses severe threats, including data theft, scams, misinformation, and system-wide disruption, all while propagating silently through the system. Our extensive experiments demonstrate that multi-agent systems are highly susceptible, even when agents do not publicly share all communications. To address this, we propose LLM Tagging, a defense mechanism that, when combined with existing safeguards, significantly mitigates infection spread. This work underscores the urgent need for advanced security measures as multi-agent LLM systems become more widely adopted.

Motivation & Objective

• To investigate the vulnerability of multi-agent systems (MAS) to LLM-to-LLM prompt injection attacks, which extend beyond single-agent threats.
• To demonstrate that malicious prompts can self-replicate across agents, leading to coordinated data exfiltration, misinformation, and system disruption.
• To evaluate the effectiveness of existing defense mechanisms—such as marking, instruction defense, and sandwiching—against this new attack surface.
• To propose and validate LLM Tagging as a defense that, when combined with other techniques, significantly reduces infection spread in MAS.
• To challenge the assumption that distributed MAS architectures are inherently more secure, revealing internal agent-to-agent compromise as a critical threat.

Proposed method

• The authors design and execute a novel attack vector called Prompt Infection, where a compromised agent propagates malicious prompts to other agents through coordinated communication.
• They simulate multi-agent systems using frameworks like LangGraph and AutoGen, enabling agents with distinct roles (e.g., memory retrieval, code execution) to collaborate.
• The attack is tested across multiple scenarios: data theft, social simulation, and tool-based command execution, using models including GPT-3.5 Turbo and GPT-4o.
• LLM Tagging is introduced as a defense mechanism that appends a unique marker to agent-generated outputs, helping downstream agents distinguish between user inputs and LLM-generated content.
• The effectiveness of LLM Tagging is evaluated in combination with other prompt-based defenses (e.g., marking, instruction defense, sandwiching), using attack success rate as the primary metric.
• Experiments are conducted using handcrafted and algorithmically generated attack prompts to assess robustness against evasion techniques.

Figure 1: Detailed Example of Prompt Infection (Data Theft). The first agent that interacts with the contaminated external document becomes compromised, extracting and propagating the infection prompt. Compromised downstream agents then execute specific instructions designed for each agent of intere

Experimental results

Research questions

• RQ1 Can malicious prompts self-replicate across interconnected LLM agents in multi-agent systems, leading to systemic compromise?
• RQ2 How does the attack success rate of Prompt Infection vary across different LLM architectures, including GPT-3.5 Turbo and GPT-4o?
• RQ3 To what extent do existing defense mechanisms—such as marking, instruction defense, and sandwiching—protect against LLM-to-LLM prompt injection?
• RQ4 Can LLM Tagging alone prevent infection, or is it only effective when combined with other defenses?
• RQ5 How does the presence of memory retrieval systems and agent collaboration affect the propagation and impact of prompt infections in social simulation environments?

Key findings

• Prompt Infection enables self-replicating malicious prompts to spread across agents, resulting in coordinated data exfiltration and system disruption, even when agents do not publicly share all communications.
• More powerful models like GPT-4o are more effective at executing attacks when compromised, due to enhanced reasoning and tool-use capabilities, increasing the systemic risk.
• The Marking + LLM Tagging defense combination reduced attack success rates to 0%, outperforming all other tested configurations.
• Instruction Defense + LLM Tagging lowered the attack success rate to 3%, while Sandwich + LLM Tagging reduced it to 16%, demonstrating the value of hybrid defense strategies.
• LLM Tagging alone reduced attack success by only 5%, indicating that standalone defenses are insufficient against sophisticated LLM-to-LLM infections.
• The attack successfully bypassed defenses using techniques like interleaving characters (e.g., _), showing that handcrafted markers can be evaded by algorithmically generated prompts.

Figure 2: Overview of Prompt Infection (Data Theft). Agents with different tools collaborate to exfiltrate data.