[Paper Review] Identifying the Risks of LM Agents with an LM-Emulated Sandbox

Yangjun Ruan, Honghua Dong+7 more|arXiv (Cornell University)|Sep 25, 2023

Multi-Agent Systems and Negotiation4 citations

TL;DR

This paper introduces ToolEmu, a language model-emulated sandbox that enables scalable, automated testing of LMs agents across diverse tools and failure scenarios. By using LMs to simulate tool execution and an LM-based evaluator to assess risks, the framework identifies 68.8% of failures as real-world valid, revealing that even the safest agent fails 23.9% of the time under high-stakes conditions.

ABSTRACT

Recent advances in Language Model (LM) agents and tool use, exemplified by applications like ChatGPT Plugins, enable a rich set of capabilities but also amplify potential risks - such as leaking private data or causing financial losses. Identifying these risks is labor-intensive, necessitating implementing the tools, setting up the environment for each test scenario manually, and finding risky cases. As tools and agents become more complex, the high cost of testing these agents will make it increasingly difficult to find high-stakes, long-tailed risks. To address these challenges, we introduce ToolEmu: a framework that uses an LM to emulate tool execution and enables the testing of LM agents against a diverse range of tools and scenarios, without manual instantiation. Alongside the emulator, we develop an LM-based automatic safety evaluator that examines agent failures and quantifies associated risks. We test both the tool emulator and evaluator through human evaluation and find that 68.8% of failures identified with ToolEmu would be valid real-world agent failures. Using our curated initial benchmark consisting of 36 high-stakes tools and 144 test cases, we provide a quantitative risk analysis of current LM agents and identify numerous failures with potentially severe outcomes. Notably, even the safest LM agent exhibits such failures 23.9% of the time according to our evaluator, underscoring the need to develop safer LM agents for real-world deployment.

Motivation & Objective

• To address the high cost and limited scalability of manually testing LM agents for real-world risks.
• To enable detection of long-tail, high-impact failures in LM agents without requiring full tool implementations or sandbox environments.
• To develop an automated, LM-based safety evaluator that quantifies risks from agent failures in emulated scenarios.
• To demonstrate the feasibility and validity of using LMs to emulate complex tools (e.g., terminal, IoT, robotics) for risk assessment.

Proposed method

• Use a large language model (e.g., GPT-4) to emulate tool execution based solely on tool specifications and inputs, bypassing the need for actual tool deployment.
• Design an LM-emulated sandbox that supports diverse tools—including future-integrated ones like IoT and robotics—enabling rapid prototyping and failure injection.
• Implement an adversarial emulator that automatically generates high-risk sandbox states to increase failure detection in red-teaming scenarios.
• Develop an LM-based safety evaluator that analyzes agent trajectories, identifies failures, and quantifies associated risks using human-validated criteria.
• Conduct human evaluation to validate the realism of detected failures, achieving 68.8% agreement with real-world agent failures.
• Curate a benchmark of 36 high-stakes toolkits and 144 test cases to enable quantitative risk analysis of current LM agents.

Experimental results

Research questions

• RQ1 Can an LM-based emulator effectively simulate diverse, high-stakes tools (e.g., terminal, financial, IoT) without requiring their actual implementation?
• RQ2 How effective is the LM-emulated sandbox in identifying realistic, high-impact failures in LM agents compared to real-world execution?
• RQ3 To what extent can an LM-based safety evaluator accurately detect and quantify risks from agent failures in long-tail scenarios?
• RQ4 What is the prevalence of severe failures in current LM agents when tested under realistic, adversarial conditions?
• RQ5 Can the framework detect failures that are both rare and high-impact, particularly in safety-critical domains like finance and robotics?

Key findings

• 68.8% of the failures identified by ToolEmu were validated as real-world valid failures through human evaluation, indicating strong realism in failure detection.
• Even the safest LM agent evaluated in the study exhibited failures in 23.9% of test cases, highlighting the persistent risk of deployment in real-world settings.
• The framework detected 7 severe failures in ChatGPT-3.5 on a terminal emulator, 6 of which were reproducible in a real bash terminal, demonstrating high fidelity.
• The adversarial emulator successfully generated high-risk scenarios that triggered failures in agents, increasing detection of edge cases.
• The LM-based safety evaluator achieved high efficiency—detecting complex failures in under 15 minutes, compared to 8 hours for manual setup in real environments.
• The benchmark of 36 toolkits and 144 test cases revealed systematic vulnerabilities across diverse domains, including financial, email, and robotic control tasks.