[논문 리뷰] The Autonomy Tax: Defense Training Breaks LLM Agents

Shuai Li, Yue Zhao|arXiv (Cornell University)|2026. 03. 19.

Adversarial Robustness in Machine Learning인용 수 0

한 줄 요약

프롬프트 주입으로부터 LLM 에이전트를 방어하기 위한 방어 훈련이 역설적으로 다단계 에이전트의 역량을 저하시켜 즉시 1단계 실패, 연쇄 타임아웃 및 공격 우회 비율 증가를 초래한다.

ABSTRACT

Large language model (LLM) agents increasingly rely on external tools (file operations, API calls, database transactions) to autonomously complete complex multi-step tasks. Practitioners deploy defense-trained models to protect against prompt injection attacks that manipulate agent behavior through malicious observations or retrieved content. We reveal a fundamental extbf{capability-alignment paradox}: defense training designed to improve safety systematically destroys agent competence while failing to prevent sophisticated attacks. Evaluating defended models against undefended baselines across 97 agent tasks and 1,000 adversarial prompts, we uncover three systematic biases unique to multi-step agents. extbf{Agent incompetence bias} manifests as immediate tool execution breakdown, with models refusing or generating invalid actions on benign tasks before observing any external content. extbf{Cascade amplification bias} causes early failures to propagate through retry loops, pushing defended models to timeout on 99\% of tasks compared to 13\% for baselines. extbf{Trigger bias} leads to paradoxical security degradation where defended models perform worse than undefended baselines while straightforward attacks bypass defenses at high rates. Root cause analysis reveals these biases stem from shortcut learning: models overfit to surface attack patterns rather than semantic threat understanding, evidenced by extreme variance in defense effectiveness across attack categories. Our findings demonstrate that current defense paradigms optimize for single-turn refusal benchmarks while rendering multi-step agents fundamentally unreliable, necessitating new approaches that preserve tool execution competence under adversarial conditions.

연구 동기 및 목표

• 다단계 LLM 에이전트에서 방어 훈련에 의해 유도된 세 가지 에이전트 특유의 실패 모드를 특성화한다.
• 표면 상의 상관관계를 이용하는 지름길 학습으로서 근본 원인을 진단한다.
• 단계-1 무능, 연쇄 동역학, 트리거 편향의 취약점을 분리하기 위한 진단 방법론과 데이터셋을 개발한다.
• 다양한 에이전트 작업에 걸쳐 다중 방어 방법을 평가하여 엔드투엔드 신뢰도 손실을 정량화한다.

제안 방법

• 다단계 에이전트에서 방어 훈련 효과를 능력 정렬의 역설로 형식화한다.
• 관찰 이전의 무능을 분리하기 위한 Step-1 실행 분석을 도입한다.
• 재시도 역학을 포착하기 위한 연쇄 실패 지표와 깊이 계층화된 완료율을 정의한다.
• 두 가지 진단 데이터셋을 설계한다: AgentDojo 기반 97-작업 벤치마크와 제어된 트리거 주입을 포함한 350샘플의 선별된 공격적-양성 세트.
• 세 가지 기본 모델(Llama-3-8B, Llama-3.1-8B, Mistral-7B)에서 세 가지 방어 구성(StruQ, SecAlign, Meta SecAlign)을 평가한다.
• 지표로 완료율(CR), 연쇄 실패율(CFR), 지름길 학습 하에서의 공격 탐지의 진짜/거짓 양성률을 보고한다.

Figure 1: Three Defense Training Biases in Multi-Step Agents. Agent Incompetence : 47–77% Step-1 failure on benign tasks (vs 3% baseline). Cascade Amplification : Timeouts increase from 13–50% to 99%. Trigger Bias : 73–86% attack bypass with 25–71% benign over-refusal. Defense training teaches surfa

실험 결과

연구 질문

• RQ1 Do defense-trained LLM agents exhibit unique multi-step failure modes not captured by single-turn benchmarks?
• RQ2 What are the dominant failure mechanisms (e.g., agent incompetence, cascade amplification, trigger bias) in defended agents?
• RQ3 Is defense training creating surface shortcuts that reduce both security and utility when facing sophisticated, shortcut-evading attacks?
• RQ4 How do defected defenses affect end-to-end agent task success across depth and retry dynamics?

주요 결과

• Defense training causes Step-1 incompetence on benign tasks for multi-step agents, with immediate refusals or invalid outputs before tool observations.
• Cascade amplification leads to dramatically higher timeouts in defended models (up to 99% CFR) compared to baseline (13–50% CFR).
• Trigger bias enables high attack bypass rates (73–86%) while simultaneously increasing false refusals on benign content (25–71% FPR).
• Defense methods exhibit substantial variance in effectiveness across attack categories, demonstrating shortcut learning rather than semantic threat understanding.
• Overall, defended agents show qualitatively worse end-to-end reliability than undefended baselines across 97 tasks and 1,000 adversarial prompts.
• A unified explanation attributes failures to shortcut learning from defense datasets correlating surface cues with labels, not semantic threat detection.

Figure 2: Immediate execution failures and utility degradation in defense-trained agents. (a) Step-1 valid action rates: Base models achieve 96.9–97.9% valid actions on the first step, while defense models drop to 22.7–53.6%, indicating immediate execution incompetence before any tool observations a