[Paper Review] AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways

Zehang Deng, Yongjian Guo+5 more|arXiv (Cornell University)|Jun 4, 2024

Network Security and Intrusion Detection4 citations

TL;DR

This survey identifies four critical knowledge gaps in AI agent security—unpredictable multi-step inputs, internal execution complexity, environmental variability, and untrusted external interactions—systematically categorizing 100+ threats and defenses. It proposes future research pathways in robust memory management, secure interaction architectures, safety benchmarks, and policy frameworks to enhance trustworthiness in LLM-based agents.

ABSTRACT

An Artificial Intelligence (AI) agent is a software entity that autonomously performs tasks or makes decisions based on pre-defined objectives and data inputs. AI agents, capable of perceiving user inputs, reasoning and planning tasks, and executing actions, have seen remarkable advancements in algorithm development and task performance. However, the security challenges they pose remain under-explored and unresolved. This survey delves into the emerging security threats faced by AI agents, categorizing them into four critical knowledge gaps: unpredictability of multi-step user inputs, complexity in internal executions, variability of operational environments, and interactions with untrusted external entities. By systematically reviewing these threats, this paper highlights both the progress made and the existing limitations in safeguarding AI agents. The insights provided aim to inspire further research into addressing the security threats associated with AI agents, thereby fostering the development of more robust and secure AI agent applications.

Motivation & Objective

• To identify and systematize the key security challenges facing LLM-based AI agents across their full lifecycle.
• To address the under-explored security vulnerabilities arising from user input unpredictability, internal complexity, environmental variability, and untrusted interactions.
• To bridge the gap in unified safety evaluation standards and policy frameworks for AI agent development and deployment.
• To inspire future research in secure agent design through systematic threat categorization and defense analysis.

Proposed method

• Categorizes AI agent security threats into four knowledge gaps: (1) unpredictable multi-step user inputs, (2) complex internal executions, (3) variable operational environments, and (4) interactions with untrusted external entities.
• Maps threats to specific agent components: perception (e.g., prompt injection), reasoning (e.g., jailbreaks), action (e.g., tool misuse), memory (e.g., poisoning), and environment (e.g., sandbox escapes).
• Reviews 100+ existing studies to classify attack surfaces and defense mechanisms across agent lifecycle stages.
• Proposes future research directions including secure memory management (e.g., AvalonBench, PoisonedRAG), optimal interaction architectures (e.g., CAMEL with dynamic permissions), and safety benchmarking (e.g., R-Judge, ToolEmu).
• Analyzes the limitations of current evaluation tools like R-Judge and ToolEmu, which are narrow in scope and lack holistic agent-level assessment.
• Advocates for standardized safety benchmarks and policy frameworks to ensure transparency, accountability, and ethical deployment of AI agents.

Experimental results

Research questions

• RQ1 What are the primary security threats arising from unpredictable, multi-step user inputs in AI agents?
• RQ2 How does internal execution complexity in LLM-based agents lead to vulnerabilities such as hallucinations or misalignment?
• RQ3 In what ways do environmental variability—ranging from development to physical environments—introduce new attack surfaces?
• RQ4 How do interactions with untrusted external entities, including other agents and tools, compromise agent integrity and confidentiality?
• RQ5 What are the critical research gaps in evaluating AI agent trustworthiness, and how can unified safety benchmarks be designed?

Key findings

• The survey identifies 100+ papers and systematically maps 100+ distinct threats across four core knowledge gaps in AI agent security.
• Prompt injection, jailbreaks, and hallucinations are major threats to perception and reasoning, often stemming from ambiguous or malicious user inputs.
• Internal execution complexity leads to risks such as misaligned behavior, unintended actions, and tool misuse, especially when reasoning and planning are not properly constrained.
• Environmental variability—including simulated, sandboxed, and physical environments—introduces threats like environment spoofing, resource exhaustion, and configuration attacks.
• Interactions with untrusted agents and tools create risks such as data leakage, cooperative/competitive threats, and memory poisoning, especially in multi-agent systems.
• Current benchmarks like R-Judge and ToolEmu are limited in scope and do not support holistic evaluation of agent trustworthiness, highlighting a critical need for unified safety standards.