[Paper Review] Adversarial Policies: Attacking Deep Reinforcement Learning

Adam Gleave, Michael D. Dennis+4 more|arXiv (Cornell University)|May 25, 2019

Adversarial Robustness in Machine LearningComputer Science33 references92 citations

TL;DR

The paper shows that adversarial policies acting in a shared environment can reliably defeat fixed victim RL policies by inducing adversarial, off-distribution observations, especially in high-dimensional settings. It analyzes why this works and explores defenses.

ABSTRACT

Deep reinforcement learning (RL) policies are known to be vulnerable to adversarial perturbations to their observations, similar to adversarial examples for classifiers. However, an attacker is not usually able to directly modify another agent's observations. This might lead one to wonder: is it possible to attack an RL agent simply by choosing an adversarial policy acting in a multi-agent environment so as to create natural observations that are adversarial? We demonstrate the existence of adversarial policies in zero-sum games between simulated humanoid robots with proprioceptive observations, against state-of-the-art victims trained via self-play to be robust to opponents. The adversarial policies reliably win against the victims but generate seemingly random and uncoordinated behavior. We find that these policies are more successful in high-dimensional environments, and induce substantially different activations in the victim policy network than when the victim plays against a normal opponent. Videos are available at https://adversarialpolicies.github.io/.

Motivation & Objective

• Introduce a physically realistic threat model where an adversary controls an opponent in a zero-sum Markov game.
• Demonstrate the existence of adversarial policies that reliably beat state-of-the-art victims trained via self-play.
• Analyze how adversarial policies manipulate observations and victim activations to cause failures.
• Investigate the role of observation dimensionality and conduct ablations to understand defense prospects.

Proposed method

• Model the victim and attacker as players in a two-player Markov game with a fixed victim policy. The attacker solves an RL problem to maximize its discounted reward given the victim’s policy embedded in the dynamics.
• Train adversarial policies using Proximal Policy Optimization (PPO) against fixed black-box victims.
• Evaluate adversaries across zero-sum simulated robotics environments with proprioceptive observations (Kick and Defend, You Shall Not Pass, Sumo Humans, Sumo Ants).
• Compare adversaries to baselines (Rand, Zero, Zoo policies) and measure win rates over time against the median victim.
• Analyze victim activations with Gaussian Mixture Models and t-SNE to understand distribution shifts caused by adversaries.

Experimental results

Research questions

• RQ1 Do adversarial policies exist in multi-agent, physically realistic RL settings where the attacker cannot directly modify the victim’s observations?
• RQ2 Can an adversarial policy outperform pretrained Zoo baselines against victims trained via self-play?
• RQ3 What mechanisms (observation manipulation, activation shift) enable adversarial policies to defeat victims, and how does observation dimensionality affect vulnerability?
• RQ4 Can defenses such as fine-tuning against adversaries mitigate attacks, and are new adversaries still able to defeat defended victims?

Key findings

• Adversarial policies reliably beat the victim policies in several environments, often with higher win rates than Zoo baselines.
• Adversaries win by creating natural, adversarial observations rather than becoming generally strong opponents, inducing off-distribution activations in the victim’s network.
• Higher observation dimensionality increases vulnerability to adversarial policies (e.g., Sumo Humans more vulnerable than Sumo Ants).
• Masking the adversary’s position can degrade normal opponents but benefits against adversaries, showing non-transitive policy interactions.
• Fine-tuning provides partial defense against a specific adversary, but new adversaries trained against the defended victim can still succeed.