[Paper Review] Robust Reinforcement Learning for Continuous Control with Model Misspecification

Daniel J. Mankowitz, Nir Levine+8 more|arXiv (Cornell University)|Jun 18, 2019

Reinforcement Learning in Robotics40 references38 citations

TL;DR

This paper introduces Robust MPO (R-MPO) and Soft Robust MPO (SRE-MPO) to optimize for worst-case returns under transition perturbations, extending MPO with robust and entropy-regularized Bellman operators, and demonstrates improved performance across nine Mujoco domains and a high-dimensional Shadow hand.

ABSTRACT

We provide a framework for incorporating robustness -- to perturbations in the transition dynamics which we refer to as model misspecification -- into continuous control Reinforcement Learning (RL) algorithms. We specifically focus on incorporating robustness into a state-of-the-art continuous control RL algorithm called Maximum a-posteriori Policy Optimization (MPO). We achieve this by learning a policy that optimizes for a worst case expected return objective and derive a corresponding robust entropy-regularized Bellman contraction operator. In addition, we introduce a less conservative, soft-robust, entropy-regularized objective with a corresponding Bellman operator. We show that both, robust and soft-robust policies, outperform their non-robust counterparts in nine Mujoco domains with environment perturbations. In addition, we show improved robust performance on a high-dimensional, simulated, dexterous robotic hand. Finally, we present multiple investigative experiments that provide a deeper insight into the robustness framework. This includes an adaptation to another continuous control RL algorithm as well as learning the uncertainty set from offline data. Performance videos can be found online at https://sites.google.com/view/robust-rl.

Motivation & Objective

• Motivate robustness to transition-dynamics perturbations (model misspecification) in continuous control RL.
• Incorporate robustness into MPO and extend to entropy-regularized objectives.
• Develop robust and soft-robust entropy-regularized Bellman operators with contraction properties.
• Empirically validate robustness across multiple Mujoco domains and a high-dimensional dexterous hand.
• Explore additional analyses such as learning uncertainty sets from offline data and adaptations to other algorithms.

Proposed method

• Derive a robust Bellman operator by replacing the standard TD error with a worst-case infimum over an uncertainty set of next states.
• Incorporate this into MPO’s policy evaluation step to learn a robust value function and derive a robust policy via a robust proposal distribution.
• Extend the operator to robust and soft-robust entropy-regularized versions and prove contraction properties.
• Instantiate Robust Entropy-regularized MPO (RE-MPO) and Soft RE-MPO (SRE-MPO) and compare to E-MPO and MPO.
• Demonstrate robustness through experiments on nine MuJoCo domains and the Shadow hand, and conduct investigative analyses (uncertainty set design, domain randomization, offline data, etc.).

Experimental results

Research questions

• RQ1 Does incorporating worst-case robustness to transition perturbations improve performance in continuous control tasks under model misspecification?
• RQ2 How do robust and soft-robust entropy-regularized objectives compare to standard MPO across diverse domains?
• RQ3 Can robustness techniques transfer to other RL algorithms and can uncertainty sets be learned from offline data?
• RQ4 What is the impact of uncertainty-set design and domain randomization on robustness performance?
• RQ5 How does robustness scale to high-dimensional, dexterous control like the Shadow hand?

Key findings

• Robust MPO (R-MPO) and Soft ROBUST MPO (SR-MPO) outperform their non-robust counterparts across nine MuJoCo domains with environment perturbations.
• The entropy-regularized versions (RE-MPO and SRE-MPO) perform at least as well as, and sometimes better than, their non-robust equivalents.
• Robust methods also show improved performance on the high-dimensional Shadow hand task compared with non-robust MPO.
• Soft-robust variants often outperform non-robust baselines, though their advantage can diminish with larger perturbations in some tasks.
• Learning the uncertainty set from offline data (DDR-MPO) can yield competitive or superior robustness as the data size increases, converging to R-MPO performance with large datasets.