[Paper Review] CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee

Tengyu Xu, Yingbin Liang+1 more|arXiv (Cornell University)|Nov 11, 2020

Reinforcement Learning in Robotics66 references22 citations

TL;DR

This paper proposes Constraint-Rectified Policy Optimization (CRPO), a novel primal approach for safe reinforcement learning that alternately optimizes the reward and rectifies constraint violations without dual variables. CRPO achieves an O(1/√T) convergence rate to the globally optimal policy and O(1/√T) constraint satisfaction error, providing the first finite-time analysis with global optimality guarantee for primal SRL algorithms.

ABSTRACT

In safe reinforcement learning (SRL) problems, an agent explores the environment to maximize an expected total reward and meanwhile avoids violation of certain constraints on a number of expected total costs. In general, such SRL problems have nonconvex objective functions subject to multiple nonconvex constraints, and hence are very challenging to solve, particularly to provide a globally optimal policy. Many popular SRL algorithms adopt a primal-dual structure which utilizes the updating of dual variables for satisfying the constraints. In contrast, we propose a primal approach, called constraint-rectified policy optimization (CRPO), which updates the policy alternatingly between objective improvement and constraint satisfaction. CRPO provides a primal-type algorithmic framework to solve SRL problems, where each policy update can take any variant of policy optimization step. To demonstrate the theoretical performance of CRPO, we adopt natural policy gradient (NPG) for each policy update step and show that CRPO achieves an $\\mathcal{O}(1/\\sqrt{T})$ convergence rate to the global optimal policy in the constrained policy set and an $\\mathcal{O}(1/\\sqrt{T})$ error bound on constraint satisfaction. This is the first finite-time analysis of primal SRL algorithms with global optimality guarantee. Our empirical results demonstrate that CRPO can outperform the existing primal-dual baseline algorithms significantly.

Motivation & Objective

• To address the lack of provable global convergence in primal-type safe reinforcement learning (SRL) algorithms.
• To develop a method that avoids the hyperparameter sensitivity of primal-dual approaches, particularly in dual variable learning rates and initialization.
• To establish a finite-time convergence rate with global optimality guarantee for a primal SRL algorithm.
• To demonstrate that a primal approach can match or exceed the performance of existing primal-dual SRL methods.

Proposed method

• CRPO alternates between unconstrained policy optimization for the reward objective and unconstrained policy minimization for violated constraints.
• It uses natural policy gradient (NPG) updates for both objective maximization and constraint rectification, avoiding dual variable updates.
• The algorithm maintains feasibility by projecting policy updates only through policy gradient steps, not through projection thresholds.
• It employs a two-phase update: first improve the policy for the reward, then immediately correct it if constraints are violated by minimizing the constraint function.
• The method avoids dual variables entirely, reducing implementation complexity and hyperparameter tuning compared to primal-dual methods.
• Theoretical analysis relies on KL divergence bounds and concentration inequalities to derive finite-time convergence rates.

Experimental results

Research questions

• RQ1 Can a primal SRL algorithm achieve global convergence to the optimal feasible policy without using dual variables?
• RQ2 What is the finite-time convergence rate of a primal SRL algorithm to the globally optimal policy?
• RQ3 Can a primal approach outperform established primal-dual SRL baselines in practice?
• RQ4 Does the absence of dual variables lead to better robustness and reduced hyperparameter tuning?
• RQ5 What is the trade-off between constraint violation and reward maximization in the proposed primal framework?

Key findings

• CRPO achieves an O(1/√T) convergence rate to the globally optimal policy in the constrained policy set.
• The algorithm ensures O(1/√T) error bound on constraint satisfaction, meaning constraint violations diminish at this rate.
• The convergence rate is established under finite-time analysis, marking the first such result for primal SRL algorithms.
• Empirical results show CRPO outperforms existing primal-dual baseline algorithms significantly in performance.
• The method avoids dual variable tuning and does not require feasible initialization, simplifying implementation.
• Theoretical analysis confirms that the algorithm maintains feasibility and convergence even with function approximation errors.