[Paper Review] Robust Adversarial Reinforcement Learning

Lerrel Pinto, James Davidson+2 more|arXiv (Cornell University)|Mar 8, 2017

Adversarial Robustness in Machine Learning29 references383 citations

TL;DR

RARL trains a protagonist under a destabilizing adversary that applies disturbances, forming a minimax objective to produce robust policies that transfer better across varying conditions.

ABSTRACT

Deep neural networks coupled with fast simulation and improved computation have led to recent successes in the field of reinforcement learning (RL). However, most current RL-based approaches fail to generalize since: (a) the gap between simulation and real world is so large that policy-learning approaches fail to transfer; (b) even if policy learning is done in real world, the data scarcity leads to failed generalization from training to test scenarios (e.g., due to different friction or object masses). Inspired from H-infinity control methods, we note that both modeling errors and differences in training and test scenarios can be viewed as extra forces/disturbances in the system. This paper proposes the idea of robust adversarial reinforcement learning (RARL), where we train an agent to operate in the presence of a destabilizing adversary that applies disturbance forces to the system. The jointly trained adversary is reinforced -- that is, it learns an optimal destabilization policy. We formulate the policy learning as a zero-sum, minimax objective function. Extensive experiments in multiple environments (InvertedPendulum, HalfCheetah, Swimmer, Hopper and Walker2d) conclusively demonstrate that our method (a) improves training stability; (b) is robust to differences in training/test conditions; and c) outperform the baseline even in the absence of the adversary.

Motivation & Objective

• Motivate robust policy learning to address sim-to-real gaps and data scarcity in real-world RL.
• Model uncertainties as adversarial disturbances applied during training.
• Jointly train a protagonist and a reinforced adversary in a zero-sum game to improve robustness.
• Demonstrate improved stability, robustness to environmental changes, and transfer to unseen settings across multiple control tasks.

Proposed method

• Formulate policy learning as a two-player zero-sum Markov game with a protagonist and an adversary.
• Use an alternating optimization procedure where the protagonist is trained with a fixed adversary, then the adversary is trained with the protagonist fixed.
• Adversary applies disturbances at predefined points to induce hard trajectories that test policy robustness.
• Adversary policy is learned to maximize disruption, effectively sampling worst-case trajectories.
• Policy updates use Trust Region Policy Optimization (TRPO) with neural network function approximators.

Experimental results

Research questions

• RQ1 Can an adversarial disturbance model improve robustness of RL policies to modeling errors and test-time variations?
• RQ2 Does jointly training a reinforced adversary lead to policies that generalize better across different masses, frictions, and initializations?
• RQ3 How does RARL performance compare to standard RL baselines (e.g., TRPO) under adversarial disturbances and altered test conditions?

Key findings

• RARL yields better mean rewards and lower variance across multiple initializations compared to TRPO on HalfCheetah, Swimmer, Hopper, and Walker2d.
• Under adversarial disturbances, RARL-trained policies remain more robust than baselines, maintaining higher performance percentile curves.
• Policies trained with RARL generalize better to changes in mass and friction than baseline policies.
• Visualization shows adversaries apply intuitive forces that destabilize the system in interpretable ways, aligning with expected physical challenges.
• Table 1 reports mean rewards (± SD) where RARL often matches or exceeds baseline across tasks (InvertedPendulum, HalfCheetah, Swimmer, Hopper, Walker2d).