[Paper Review] Delving into adversarial attacks on deep policies

Jernej Kos, Dawn Song|arXiv (Cornell University)|May 18, 2017

Adversarial Robustness in Machine Learning4 references121 citations

TL;DR

This paper studies adversarial attacks on deep reinforcement learning policies, comparing adversarial perturbations to random noise, using value-function guidance to optimize attack timing, and evaluating re-training for resilience.

ABSTRACT

Adversarial examples have been shown to exist for a variety of deep learning architectures. Deep reinforcement learning has shown promising results on training agent policies directly on raw inputs such as image pixels. In this paper we present a novel study into adversarial attacks on deep reinforcement learning polices. We compare the effectiveness of the attacks using adversarial examples vs. random noise. We present a novel method for reducing the number of times adversarial examples need to be injected for a successful attack, based on the value function. We further explore how re-training on random noise and FGSM perturbations affects the resilience against adversarial examples.

Motivation & Objective

• Assess the effectiveness of adversarial examples versus random noise on deep RL policies.
• Evaluate whether injecting perturbations less frequently can still yield effective attacks.
• Investigate using the value function to time adversarial perturbations.
• Examine how re-training with noise or adversarial perturbations affects resilience and transfer to other perturbation magnitudes.
• Provide preliminary insights into how DRL policies respond to adversarial inputs.

Proposed method

• Use a TensorFlow A3C implementation on Atari Pong with 42x42 grayscale inputs.
• Generate FGSM perturbations using cross-entropy loss between policy logits and the argmax action.
• Compare FGSM attacks to uniform random noise with beta-based magnitude.
• Test three perturbation timing strategies: every N frames, recompute every N frames, and value-function guided injection.
• Evaluate re-training the agent in noisy environments and assess transfer to different perturbation magnitudes and types.
• Visualize policy-space perturbations to analyze decision boundary fragmentation.

Experimental results

Research questions

• RQ1 How do adversarial examples compare to random noise in attacking DRL policies?
• RQ2 Can perturbations be effectively injected less frequently without losing attack efficacy?
• RQ3 Does using the value function to guide perturbation timing improve attack success?
• RQ4 Does re-training with noise or FGSM perturbations increase resilience to FGSM and other perturbations?
• RQ5 How do perturbations affect the policy’s decision boundary and action mappings in image space?

Key findings

• FGSM adversarial perturbations are orders of magnitude more effective than random noise at degrading DRL policy performance at low perturbation magnitudes.
• Guided injection using the value function allows effective attacks while perturbations are injected in only a fraction of frames.
• Recomputing perturbations every N frames and reusing them in intermediate frames can match the effectiveness of injecting on every frame.
• Re-training with random noise or FGSM perturbations can improve resilience to FGSM attacks, with some transfer to different perturbation magnitudes.
• Re-trained policies may still be vulnerable to other attack methods and the policy action space can remain fragmented even after re-training.