[Paper Review] Adversarial Attacks on Neural Network Policies

Sandy H. Huang, Nicolas Papernot+3 more|arXiv (Cornell University)|Feb 8, 2017

Adversarial Robustness in Machine Learning68 citations

TL;DR

This paper demonstrates that neural network policies in deep reinforcement learning are vulnerable to adversarial examples, under both white-box and black-box settings, across multiple algorithms and Atari tasks.

ABSTRACT

Machine learning classifiers are known to be vulnerable to inputs maliciously constructed by adversaries to force misclassification. Such adversarial examples have been extensively studied in the context of computer vision applications. In this work, we show adversarial attacks are also effective when targeting neural network policies in reinforcement learning. Specifically, we show existing adversarial example crafting techniques can be used to significantly degrade test-time performance of trained policies. Our threat model considers adversaries capable of introducing small perturbations to the raw input of the policy. We characterize the degree of vulnerability across tasks and training algorithms, for a subclass of adversarial-example attacks in white-box and black-box settings. Regardless of the learned task or training algorithm, we observe a significant drop in performance, even with small adversarial perturbations that do not interfere with human perception. Videos are available at http://rll.berkeley.edu/adversarial.

Motivation & Objective

• Characterize how adversarial examples affect neural network policies trained with deep RL across different learning algorithms.
• Investigate white-box and black-box attack scenarios on policies trained with DQN, TRPO, and A3C.
• Assess transferability of adversarial examples across policies and across training algorithms.

Proposed method

• Apply Fast Gradient Sign Method (FGSM) to generate adversarial perturbations under l_infty, l_2, and l_1 norms.
• Treat policy outputs as probability distributions and use cross-entropy loss aligned with the highest-weighted action to compute perturbations.
• Evaluate attacks on four Atari games (Chopper Command, Pong, Seaquest, Space Invaders) trained with three algorithms (DQN, TRPO, A3C).
• Compare white-box attacks (full access to target policy) with black-box attacks via transferability across policies and across algorithms.

Experimental results

Research questions

• RQ1 How effective are adversarial examples at degrading test-time performance of neural network policies across different deep RL algorithms?
• RQ2 Do adversarial perturbations transfer between policies trained for the same task but with different architectures or training methods?
• RQ3 How does adversarial effectiveness differ between white-box and black-box threat models in deep RL policies?
• RQ4 What is the impact of using different norm constraints (l_infty, l_2, l_1) on the strength of adversarial attacks against RL policies?

Key findings

• Neural network policies across DQN, TRPO, and A3C are vulnerable to adversarial inputs, with significant performance drops.
• TRPO and A3C policies show relative resilience compared to DQN under adversarial perturbations.
• l_infty FGSM with small epsilon can reduce performance by 50% or more on some tasks, while l_1 perturbations can cause large performance drops by changing a few pixels.
• Adversarial perturbations trained for one policy can transfer to others solving the same task, demonstrating transferability.
• Transferability is stronger across policies of the same algorithm and weaker across different algorithms, but still impactful at larger perturbation magnitudes.