[Paper Review] Action Robust Reinforcement Learning and Applications in Continuous Control

Chen Tessler, Yonathan Efroni+1 more|arXiv (Cornell University)|Jan 26, 2019

Adversarial Robustness in Machine Learning43 references65 citations

TL;DR

The paper formalizes two action-robust criteria (PR-MDP and NR-MDP) for RL under action uncertainty, provides theoretical guarantees for PR-MDP, and demonstrates improved robustness and performance in continuous control tasks via deep RL extensions and AR-DDPG.

ABSTRACT

A policy is said to be robust if it maximizes the reward while considering a bad, or even adversarial, model. In this work we formalize two new criteria of robustness to action uncertainty. Specifically, we consider two scenarios in which the agent attempts to perform an action $a$, and (i) with probability $α$, an alternative adversarial action $\bar a$ is taken, or (ii) an adversary adds a perturbation to the selected action in the case of continuous action space. We show that our criteria are related to common forms of uncertainty in robotics domains, such as the occurrence of abrupt forces, and suggest algorithms in the tabular case. Building on the suggested algorithms, we generalize our approach to deep reinforcement learning (DRL) and provide extensive experiments in the various MuJoCo domains. Our experiments show that not only does our approach produce robust policies, but it also improves the performance in the absence of perturbations. This generalization indicates that action-robustness can be thought of as implicit regularization in RL problems.

Motivation & Objective

• Motivate robustness to action uncertainty in RL for real-world continuous control tasks.
• Introduce two robustness criteria (PR-MDP and NR-MDP) that model adversarial or perturbed actions.
• Establish theoretical properties and duality for PR-MDP and NR-MDP and derive policy iteration schemes.
• Extend to deep RL and validate with extensive MuJoCo experiments showing robust performance and generalization.

Proposed method

• Define PR-MDP as a zero-sum game where with probability alpha an adversary takes a worse action and derive a max-min objective.
• Define NR-MDP where an adversary adds a perturbation to the chosen action and analyze its properties.
• Prove existence of deterministic stationary optimal policy for PR-MDP and strong duality.
• Propose Policy Iteration schemes for PR-MDP (Probabilistic Robust PI and Soft PR-PI) and discuss their convergence.
• Develop Action-Robust DDPG (AR-DDPG) by integrating actor and adversary networks and a critic for joint actions in continuous control.
• Provide gradient-based update rules for actor and adversary in PR- and NR-MDP settings (Proposition 5).

Experimental results

Research questions

• RQ1 How can action uncertainty be formally modeled in RL to produce robust policies?
• RQ2 What are the theoretical properties (existence, duality, convergence) of PR-MDP and NR-MDP frameworks?
• RQ3 Can these action-robust criteria be extended to deep RL and still yield robust, or even improved, performance in practice?
• RQ4 Do action-robust methods transfer across MuJoCo domains and under varying perturbation strengths?
• RQ5 What is the practical behavior and trade-off between probabilistic action robustness (PR-MDP) and noisy action robustness (NR-MDP) in continuous control?

Key findings

• PR-MDP has a well-defined value and admits a deterministic stationary optimal policy with strong duality.
• NR-MDP generally requires stochastic policies for optimum, and strong duality may not hold within deterministic policy classes.
• Soft PR-PI converges to the Nash-equilibrium value under error-free conditions, and PR-PI converges similarly; both leverage a mix of agent and adversary updates.
• AR-DDPG, a DRL instantiation, can learn robust policies in MuJoCo domains and often improves performance even without perturbations, suggesting action robustness acts as implicit regularization.
• Empirical results indicate hyperparameters (e.g., alpha) influence robustness and performance, with PR-MDP showing more stable behavior across domains than NR-MDP in many settings.