[Paper Review] Tactics of Adversarial Attack on Deep Reinforcement Learning Agents

Yen-Chen Lin, Zhang-Wei Hong+4 more|arXiv (Cornell University)|Mar 8, 2017

Adversarial Robustness in Machine LearningComputer Science96 citations

TL;DR

The paper introduces two adversarial attack tactics for deep RL agents: strategically-timed attacks that perturb observations at a subset of steps, and enchanting attacks that plan perturbations to steer the agent to a target state, evaluated on A3C and DQN across five Atari games.

ABSTRACT

We introduce two tactics to attack agents trained by deep reinforcement learning algorithms using adversarial examples, namely the strategically-timed attack and the enchanting attack. In the strategically-timed attack, the adversary aims at minimizing the agent's reward by only attacking the agent at a small subset of time steps in an episode. Limiting the attack activity to this subset helps prevent detection of the attack by the agent. We propose a novel method to determine when an adversarial example should be crafted and applied. In the enchanting attack, the adversary aims at luring the agent to a designated target state. This is achieved by combining a generative model and a planning algorithm: while the generative model predicts the future states, the planning algorithm generates a preferred sequence of actions for luring the agent. A sequence of adversarial examples is then crafted to lure the agent to take the preferred sequence of actions. We apply the two tactics to the agents trained by the state-of-the-art deep reinforcement learning algorithm including DQN and A3C. In 5 Atari games, our strategically timed attack reduces as much reward as the uniform attack (i.e., attacking at every time step) does by attacking the agent 4 times less often. Our enchanting attack lures the agent toward designated target states with a more than 70% success rate. Videos are available at http://yenchenlin.me/adversarial_attack_RL/

Motivation & Objective

• Understand vulnerability of deep RL agents to adversarial perturbations.
• Develop tactics that minimize perturbations while reducing agent rewards.
• Demonstrate effectiveness of attacks on state-of-the-art Deep RL algorithms (A3C, DQN).
• Explore planning-based attacks to steer agents toward designated states.

Proposed method

• Define strategically-timed attack using a relative action preference function to decide when to perturb.
• Craft perturbations with Carlini & Wagner method to flip the agent’s most preferred action to the least preferred one.
• Limit total attacks by a budget Γ and evaluate reward impact versus uniform attacks.
• Introduce enchanting attack combining a video-prediction model and a planning algorithm to lure the agent to a target state over H steps.
• Use a future-state predictor M to estimate s_{t+H}^M = M(s_t, A_{t:t+H}) and a sampling-based cross-entropy method to plan action sequences A_{t:t+H}.
• Evaluate on Atari games (MsPacman, Pong, Seaquest, Qbert, ChopperCommand) with A3C and DQN.

Experimental results

Research questions

• RQ1 Can deep RL agents trained with DQN and A3C be effectively attacked using minimally perturbed observations without triggering easy detection?
• RQ2 How effective are strategically-timed attacks compared to uniform attacks in reducing accumulated rewards?
• RQ3 Can a planning-based enchanting attack reliably steer an agent to a designated target state, and under what conditions?
• RQ4 What defense considerations emerge for robustness against these two adversarial tactics?

Key findings

• Strategically-timed attacks can match the reward reduction of uniform attacks while perturbing observations at roughly 25% of time steps on average.
• DQN agents tend to be more vulnerable to strategically-timed attacks than A3C in most games examined.
• Enchanting attacks achieve more than 70% success in luring agents toward target states for several settings and games.
• The enchanting attack is less effective in environments with high stochasticity (e.g., multiple random enemies) due to prediction model inaccuracies.
• The study demonstrates two novel attack vectors against state-of-the-art Deep RL agents and discusses potential defenses.