[Paper Review] Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learning

Jianwen Sun, Tianwei Zhang+5 more|arXiv (Cornell University)|May 14, 2020

Adversarial Robustness in Machine Learning39 references4 citations

TL;DR

This paper proposes two novel adversarial attack techniques—Critical Point Attack and Antagonist Attack—for stealthy and efficient evasion of Deep Reinforcement Learning (DRL) agents. By leveraging predictive modeling of future states and damage-aware optimization, the methods identify minimal critical intervention points, achieving successful attacks in just 1–5 steps across diverse environments like Atari, TORCS, and MuJoCo, significantly outperforming prior methods that require 20–80 steps.

ABSTRACT

Adversarial attacks against conventional Deep Learning (DL) systems and algorithms have been widely studied, and various defenses were proposed. However, the possibility and feasibility of such attacks against Deep Reinforcement Learning (DRL) are less explored. As DRL has achieved great success in various complex tasks, designing effective adversarial attacks is an indispensable prerequisite towards building robust DRL algorithms. In this paper, we introduce two novel adversarial attack techniques to \emph{stealthily} and \emph{efficiently} attack the DRL agents. These two techniques enable an adversary to inject adversarial samples in a minimal set of critical moments while causing the most severe damage to the agent. The first technique is the \emph{critical point attack}: the adversary builds a model to predict the future environmental states and agent's actions, assesses the damage of each possible attack strategy, and selects the optimal one. The second technique is the \emph{antagonist attack}: the adversary automatically learns a domain-agnostic model to discover the critical moments of attacking the agent in an episode. Experimental results demonstrate the effectiveness of our techniques. Specifically, to successfully attack the DRL agent, our critical point technique only requires 1 (TORCS) or 2 (Atari Pong and Breakout) steps, and the antagonist technique needs fewer than 5 steps (4 Mujoco tasks), which are significant improvements over state-of-the-art methods.

Motivation & Objective

• To address the lack of effective, stealthy adversarial attacks against Deep Reinforcement Learning (DRL) agents.
• To identify minimal, high-impact intervention points in DRL trajectories that maximize damage to the agent’s end goal.
• To develop generic, efficient attack strategies that require fewer perturbation steps than existing methods.
• To improve attack stealthiness by minimizing the number of modified time steps while maximizing disruption to the agent’s performance.
• To evaluate the proposed methods across diverse DRL environments and algorithms, including continuous and discrete action spaces.

Proposed method

• The Critical Point Attack uses a domain-specific model to predict future states and actions, and evaluates attack strategies using a Damage Awareness Metric based on the agent’s end goal.
• It selects the optimal attack strategy by simulating the impact of perturbations on long-term performance, ensuring minimal steps with maximum damage.
• The Antagonist Attack trains a domain-agnostic model to autonomously detect critical moments for intervention using the agent’s reward function as an optimization objective.
• This model learns to predict when and how to inject perturbations to steer the agent toward failure, without requiring prior knowledge of the environment.
• Both methods use small, imperceptible perturbations to environment states, ensuring stealthiness while maximizing disruption to the agent’s policy.
• The Damage Awareness Metric quantifies the long-term impact of each attack strategy, enabling the adversary to select the most effective intervention points.

Experimental results

Research questions

• RQ1 Can adversarial attacks be designed to be both stealthy and highly effective against DRL agents by minimizing the number of intervention steps?
• RQ2 Can a damage-aware metric based on the agent’s end goal improve the precision and efficiency of adversarial attacks in DRL?
• RQ3 Can a domain-agnostic antagonist model automatically identify optimal attack moments without prior environment-specific knowledge?
• RQ4 How do the proposed attacks compare in efficiency and effectiveness to state-of-the-art methods like strategically-timed and enchanting attacks?
• RQ5 Can the proposed techniques be generalized across different DRL algorithms and environments, including those with continuous action spaces?

Key findings

• The Critical Point Attack achieved successful attacks in just 1 step for TORCS and 2 steps for Atari Pong and Breakout, significantly fewer than the 80 steps required by the strategically-timed attack.
• The Antagonist Attack required fewer than 5 steps across 4 MuJoCo environments—2 steps for InvertedPendulum, 1 for Hopper, 5 for HalfCheetah, and 3 for Walker2d—demonstrating high efficiency.
• In TORCS, the optimal attack perturbation size (Δ) was found to be approximately 0.82, which consistently caused collisions or loss of control.
• The Antagonist Attack outperformed the strategically-timed attack by requiring only 3 steps for Pong and Breakout, compared to 6 and 5 steps respectively.
• The Antagonist Attack showed strategic targeting behavior, preferring to attack when the agent was near corners (steps 105–135) in TORCS, where instability increases attack effectiveness.
• The Antagonist model learned to avoid attacking during high-reward, stable phases (e.g., steps 65–95 in TORCS), where the agent’s action preference is strong, confirming its strategic awareness.