[Paper Review] Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification

Daniel J. Mankowitz, Dan A. Calian+6 more|arXiv (Cornell University)|Oct 20, 2020

Reinforcement Learning in Robotics29 references4 citations

TL;DR

This paper proposes R3C and RC reinforcement learning objectives to address constrained model misspecification in continuous control, where policies must remain robust to unseen state perturbations while satisfying constraints. By formulating a Robust Constrained MDP and defining corresponding Bellman operators, the method ensures convergence and improves constraint satisfaction and return robustness across six MuJoCo tasks compared to baselines.

ABSTRACT

Many real-world physical control systems are required to satisfy constraints upon deployment. Furthermore, real-world systems are often subject to effects such as non-stationarity, wear-and-tear, uncalibrated sensors and so on. Such effects effectively perturb the system dynamics and can cause a policy trained successfully in one domain to perform poorly when deployed to a perturbed version of the same domain. This can affect a policy's ability to maximize future rewards as well as the extent to which it satisfies constraints. We refer to this as constrained model misspecification. We present an algorithm that mitigates this form of misspecification, and showcase its performance in multiple simulated Mujoco tasks from the Real World Reinforcement Learning (RWRL) suite.

Motivation & Objective

• To address the challenge of constrained model misspecification (CMM), where policies trained in a nominal environment fail under unseen state perturbations during deployment.
• To ensure that policies remain robust to both performance degradation and constraint violations when deployed in perturbed environments.
• To develop a unified RL framework that simultaneously optimizes for return and constraint satisfaction under uncertainty.
• To extend state-of-the-art continuous control algorithms (D4PG, DMPO) with robustness to model perturbations via new value function and Bellman operator definitions.

Proposed method

• Proposes a Robust Constrained MDP (RC-MDP) formulation that models uncertainty in state transitions as an ambiguity set over transition probabilities.
• Defines two new RL objectives: Robust Return Robust Constraint (R3C) and Robust Constraint (RC), which optimize for worst-case return and constraint satisfaction across perturbed environments.
• Introduces R3C and RC Bellman operators that ensure convergence to fixed points during policy evaluation in actor-critic algorithms.
• Implements algorithmic variants of R3C and RC on top of D4PG and DMPO, integrating robust value function updates into the policy evaluation step.
• Uses Lagrangian relaxation to enforce constraints, with dynamic adjustment of the Lagrange multiplier to balance constraint violation and return maximization.
• Employs a holdout set of perturbed environments to evaluate robustness, simulating real-world deployment under unknown perturbations.

Experimental results

Research questions

• RQ1 Can RL agents be trained to maintain constraint satisfaction under state perturbations not seen during training?
• RQ2 How does robust optimization of return and constraint satisfaction affect learning stability and performance in the presence of model misspecification?
• RQ3 Do the proposed R3C and RC objectives lead to policies that generalize better to unseen perturbed environments compared to standard constrained RL?
• RQ4 What is the impact of robustness on constraint conservatism—does it lead to overly cautious behavior?
• RQ5 How do the Lagrange multipliers behave during training when optimizing for robust constraint satisfaction?

Key findings

• The R3C-DMPO and RC-DMPO algorithms outperform all baselines in maintaining constraint satisfaction on both nominal and holdout perturbed environments across six MuJoCo tasks.
• Robust variants (R3C-DMPO, RC-DMPO) achieve significantly better constraint return (J_C^π) on holdout sets, demonstrating improved robustness to unseen perturbations.
• Non-robust baselines like C-DMPO and R-DMPO fail to satisfy constraints on holdout environments despite success on the nominal training environment.
• The Lagrange multiplier remains non-zero for robust algorithms even when constraints are satisfied in the nominal environment, indicating active constraint enforcement under uncertainty.
• Learning curves show that robust algorithms converge to policies that are more conservative in constraint handling, with constraint returns consistently below the threshold β, suggesting potential over-conservatism.
• The proposed Bellman operators for R3C and RC converge to fixed points, validating the theoretical foundation of the method.