[Paper Review] Characterizing Attacks on Deep Reinforcement Learning

Xinlei Pan, Chaowei Xiao+9 more|arXiv (Cornell University)|Jul 21, 2019

Adversarial Robustness in Machine LearningComputer Science32 references52 citations

TL;DR

This paper develops realistic, efficient black-box and online adversarial attacks on deep RL systems, including observations and environment dynamics, and validates them in simulation and on real robots.

ABSTRACT

Recent studies show that Deep Reinforcement Learning (DRL) models are vulnerable to adversarial attacks, which attack DRL models by adding small perturbations to the observations. However, some attacks assume full availability of the victim model, and some require a huge amount of computation, making them less feasible for real world applications. In this work, we make further explorations of the vulnerabilities of DRL by studying other aspects of attacks on DRL using realistic and efficient attacks. First, we adapt and propose efficient black-box attacks when we do not have access to DRL model parameters. Second, to address the high computational demands of existing attacks, we introduce efficient online sequential attacks that exploit temporal consistency across consecutive steps. Third, we explore the possibility of an attacker perturbing other aspects in the DRL setting, such as the environment dynamics. Finally, to account for imperfections in how an attacker would inject perturbations in the physical world, we devise a method for generating a robust physical perturbations to be printed. The attack is evaluated on a real-world robot under various conditions. We conduct extensive experiments both in simulation such as Atari games, robotics and autonomous driving, and on real-world robotics, to compare the effectiveness of the proposed attacks with baseline approaches. To the best of our knowledge, we are the first to apply adversarial attacks on DRL systems to physical robots.

Motivation & Objective

• Motivate and understand DRL vulnerabilities under realistic attack settings.
• Develop efficient black-box adversarial attacks that do not require access to victim model parameters.
• Leverage temporal structure of MDPs to create high-throughput online attacks.
• Explore perturbations to environment dynamics and physical-world robustness of attacks.
• Demonstrate feasibility of adversarial attacks on real robots with robust physical perturbations.

Proposed method

• Adapt and improve FD-based black-box attacks with adaptive sampling to reduce gradient estimation cost.
• Propose online sequential attacks that exploit temporal consistency to attack multiple frames with a single perturbation.
• Introduce frame-selection strategies to identify important frames for attack generation.
• Develop attacks targeting environment transition dynamics via random search and RL-based dynamics search.
• Extend attacks to physical robots by generating robust printed adversarial patches and evaluating under real-world conditions.

Experimental results

Research questions

• RQ1 Can black-box attacks match or exceed white-box/black-box baselines in DRL settings?
• RQ2 Does adaptive sampling (SFD) improve gradient estimation efficiency over standard finite-difference?
• RQ3 Do online sequential attacks improve attack throughput and effectiveness over per-frame attacks?
• RQ4 Are perturbations to environment dynamics feasible and effective against DRL agents?
• RQ5 Can adversarial patches be robust in physical robot deployments?

Key findings

• Black-box attacks using adaptive sampling and finite differences can be effective without model architecture or parameter access.
• Online sequential attacks achieve high throughput by perturbing a small subset of frames and applying perturbations to subsequent frames.
• Attacks on environment dynamics via random search and RL-based methods can degrade agent performance beyond baselines.
• Physical adversarial patches can be crafted to be robust against printing and viewpoint variations in a robotic navigation task.
• Experiments span Atari games, MuJoCo control tasks, TORCS driving simulation, and real robot experiments, comparing to white-box and baseline black-box methods.