[Paper Review] Robust Reinforcement Learning on State Observations with Learned Optimal Adversary

Huan Zhang, Hongge Chen+2 more|arXiv (Cornell University)|Jan 21, 2021

Adversarial Robustness in Machine LearningComputer Science58 references46 citations

TL;DR

The paper introduces ATLA, an alternating training framework that learns a strong adversary online to perturb state observations and trains robust DRL agents, showing state-of-the-art robustness against strong adversaries in continuous control tasks.

ABSTRACT

We study the robustness of reinforcement learning (RL) with adversarially perturbed state observations, which aligns with the setting of many adversarial attacks to deep reinforcement learning (DRL) and is also important for rolling out real-world RL agent under unpredictable sensing noise. With a fixed agent policy, we demonstrate that an optimal adversary to perturb state observations can be found, which is guaranteed to obtain the worst case agent reward. For DRL settings, this leads to a novel empirical adversarial attack to RL agents via a learned adversary that is much stronger than previous ones. To enhance the robustness of an agent, we propose a framework of alternating training with learned adversaries (ATLA), which trains an adversary online together with the agent using policy gradient following the optimal adversarial attack framework. Additionally, inspired by the analysis of state-adversarial Markov decision process (SA-MDP), we show that past states and actions (history) can be useful for learning a robust agent, and we empirically find a LSTM based policy can be more robust under adversaries. Empirical evaluations on a few continuous control environments show that ATLA achieves state-of-the-art performance under strong adversaries. Our code is available at https://github.com/huanzhang12/ATLA_robust_RL.

Motivation & Objective

• Motivate robustness of RL when state observations are adversarially perturbed.
• Characterize optimal adversaries under SA-MDP framework and show they can be learned online.
• Propose ATLA to jointly train a powerful adversary and a robust agent.
• Explore the role of history (memory) in robust policy learning and evaluate on continuous control tasks.

Proposed method

• Model state observation perturbations as SA-MDP, deriving optimal adversary given a fixed policy.
• Cast optimal adversary learning as a DRL problem and train via policy gradient (PPO).
• Propose Algorithm 1 to train an online learned adversary with projection onto the perturbation set B(s).
• Propose Algorithm 2 (ATLA) to alternate updates between adversary optimization and agent optimization.
• Use LSTM-based policies to exploit history for robustness under perturbations.
• Provide theoretical links between SA-MDP and POMDP when adversaries are fixed.

Experimental results

Research questions

• RQ1 How can we compute or approximate an optimal adversary that minimizes an agent's return under bounded observation perturbations?
• RQ2 Does alternating training with a learned adversary improve robustness of DRL agents compared to prior adversarial training?
• RQ3 Is history (memory) beneficial for robust RL under state perturbations, and can recurrent policies help?
• RQ4 How does ATLA compare to existing robustness methods on continuous control tasks under strong adversarial perturbations?

Key findings

• The learned adversary can produce significantly stronger perturbations than prior attacks, reducing agent rewards more effectively.
• ATLA improves robustness and achieves state-of-the-art performance under strong adversaries across several continuous control tasks.
• LSTM-based policies show greater robustness than feedforward policies in SA-MDP/ATLA settings.
• ATLA provides robustness benefits that can complement explicit regularization on function approximators.
• Optimal adversary learning does not require access to policy gradients of the agent (black-box adversary).
• Empirical results show ATLA outperforming explicit regularization methods in several environments.