[Paper Review] First Order Constrained Optimization in Policy Space

Yiming Zhang, Quan Vuong+1 more|arXiv (Cornell University)|Feb 16, 2020

Reinforcement Learning in Robotics47 references52 citations

TL;DR

FOCOPS is a first-order constrained reinforcement learning method that optimizes return while satisfying a cost constraint by solving a nonparametric update in policy space and then projecting to a parametric policy.

ABSTRACT

In reinforcement learning, an agent attempts to learn high-performing behaviors through interacting with the environment, such behaviors are often quantified in the form of a reward function. However some aspects of behavior-such as ones which are deemed unsafe and to be avoided-are best captured through constraints. We propose a novel approach called First Order Constrained Optimization in Policy Space (FOCOPS) which maximizes an agent's overall reward while ensuring the agent satisfies a set of cost constraints. Using data generated from the current policy, FOCOPS first finds the optimal update policy by solving a constrained optimization problem in the nonparameterized policy space. FOCOPS then projects the update policy back into the parametric policy space. Our approach has an approximate upper bound for worst-case constraint violation throughout training and is first-order in nature therefore simple to implement. We provide empirical evidence that our simple approach achieves better performance on a set of constrained robotics locomotive tasks.

Motivation & Objective

• Motivate safety in reinforcement learning via CMDPs and constrained policy optimization.
• Introduce a two-step method that first finds a nonparametric optimal update and then projects it into a parametric policy space.
• Provide theoretical guarantees on approximate worst-case constraint violations during training.
• Demonstrate empirical improvements over state-of-the-art constrained RL methods on high-dimensional continuous control tasks.

Proposed method

• Formulate constrained RL as a CMDP with a single cost constraint for simplicity.
• In step one, solve the constrained optimization in the nonparameterized policy space to obtain an optimal update policy π* (Theorem 1 gives the near-closed form π*(a|s) ∝ π_θk(a|s) exp((A^{π_θk}(s,a) - ν A_C^{π_θk}(s,a))/λ) with λ,ν determined by a dual problem.
• Project π* back to the parametric policy space by minimizing the KL-divergence loss L(θ)=E_{s∼d^{π_θk}}[D_KL(π_θ(·|s) ∥ π*(·|s))].
• Use first-order gradient updates to minimize L(θ); derive gradient form (Corollary 1) that combines the policy KL term with a cost-aware advantage term.
• Update ν via a primal-dual-like rule to enforce the cost constraint (Equation 14) and employ an acceptance indicator to keep updates within the trust region (Equation 15).
• Avoid second-order Fisher information matrix inversions and rely on first-order methods for simplicity and stability; provide an empirical setup comparing to CPO and Lagrangian baselines.

Experimental results

Research questions

• RQ1 How can constrained policy optimization be efficiently implemented using only first-order methods?
• RQ2 Can solving a constrained update in the nonparametric policy space and projecting back yield improved performance with approximate constraint satisfaction?
• RQ3 What are the empirical performance and constraint-satisfaction characteristics of FOCOPS on high-dimensional continuous control tasks?
• RQ4 How do hyperparameters like λ and the maximum ν influence performance and constraint adherence?
• RQ5 How does FOCOPS compare to CPO and Lagrangian-based methods across varied robotics tasks?

Key findings

• FOCOPS achieves higher or competitive rewards while maintaining approximate constraint satisfaction across multiple MuJoCo tasks with speed limits.
• Compared to CPO and Lagrangian baselines, FOCOPS often outperforms in reward while keeping costs within the constraint, and TRPO-L can violate constraints on some tasks.
• The method is simple to implement, being fully first-order, and shows robustness to hyperparameter choices such as ν_max and λ.
• Generalization experiments indicate FOCOPS often outperforms other constrained methods on unseen seeds for most tasks except Hopper-v3.
• Sensitivity analysis shows performance is not highly sensitive to ν_max (even ∞) and λ around a reasonable value yields good results.