[Paper Review] Robust Deep Reinforcement Learning with Adversarial Attacks

Anay Pattanaik, Zhenyi Tang+3 more|arXiv (Cornell University)|Dec 11, 2017

Adversarial Robustness in Machine Learning16 references119 citations

TL;DR

The paper designs adversarial attacks for DRL to expose robustness weaknesses, then uses adversarial training to improve DRL robustness to parameter variations across Cart-pole, Mountain Car, Hopper, and Half Cheetah environments.

ABSTRACT

This paper proposes adversarial attacks for Reinforcement Learning (RL) and then improves the robustness of Deep Reinforcement Learning algorithms (DRL) to parameter uncertainties with the help of these attacks. We show that even a naively engineered attack successfully degrades the performance of DRL algorithm. We further improve the attack using gradient information of an engineered loss function which leads to further degradation in performance. These attacks are then leveraged during training to improve the robustness of RL within robust control framework. We show that this adversarial training of DRL algorithms like Deep Double Q learning and Deep Deterministic Policy Gradients leads to significant increase in robustness to parameter variations for RL benchmarks such as Cart-pole, Mountain Car, Hopper and Half Cheetah environment.

Motivation & Objective

• Motivate robustness concerns for DRL in robotics and safety-critical settings due to parameter and environmental variations.
• Propose adversarial attacks that degrade DRL performance by exploiting the agent's value function.
• Develop adversarial training within a robust control framework to improve DRL robustness across a range of parameters.
• Demonstrate robustness improvements for DRL algorithms like DDQN and DDPG on standard benchmarks.

Proposed method

• Define adversarial attack for value-function based RL as perturbations that increase the probability of taking the worst action in a state.
• Develop naive and gradient-based attacks that perturb observations within an l2-norm bound to fool DRL agents.
• Introduce a gradient-based attack that minimizes a cross-entropy-like objective derived from the policy distribution to locate worst-case states.
• Use adversarial perturbations during training to realize robust adversarial training, inspired by robust control (CVaR) objectives.
• Describe two training algorithms (Adv train) that apply gradient-based adversarial perturbations during DRQN/DDPG training to achieve robustness.
• Discuss equivalence to robust control where adversary explores worst-case parameter variations to generate hard trajectories.

Experimental results

Research questions

• RQ1 How vulnerable are DRL agents (DDQN and DDPG) to adversarial perturbations of observations?
• RQ2 Can adversarial perturbations be leveraged to train DRL agents that remain robust under a wide range of parameter variations?
• RQ3 Do gradient-based adversarial attacks outperform naive or SGD-based attacks in this RL setting?
• RQ4 What is the effect of adversarial training on performance in continuous control benchmarks like Cart-Pole, Mountain Car, Hopper, and Half-Cheetah?

Key findings

• Gradient-based adversarial attacks degrade DRL performance more effectively than naive sampling and SGD-based methods.
• RBF-based Q-learning shows more robustness to adversarial perturbations than DDQN, likely due to smoother function approximation.
• Adversarial training using the proposed attacks yields significant robustness improvements across Cart-Pole, Mountain Car, Hopper, and Half-Cheetah under parameter variations.
• Adversarially trained DDQN and DDPG achieve higher average returns over a wide range of parameter settings compared to vanilla DRL baselines.
• The attacks reveal robustness issues in prevalent DRL algorithms and show that linearly parameterized RL can be more robust to such attacks.