[Paper Review] Accelerating Safe Reinforcement Learning with Constraint-mismatched Policies

Tsung-Yen Yang, Justinian Rosca+2 more|arXiv (Cornell University)|Jun 20, 2020

Reinforcement Learning in Robotics49 references4 citations

TL;DR

This paper proposes SPACE, a safe reinforcement learning algorithm that accelerates learning from sub-optimal or constraint-violating baseline policies by iteratively optimizing for task reward, maintaining proximity to the baseline policy, and projecting onto a constraint-satisfying policy set. The method achieves 40% higher average return and 10× fewer constraint violations than state-of-the-art baselines across five control tasks, including safety and fairness constraints.

ABSTRACT

We consider the problem of reinforcement learning when provided with (1) a baseline control policy and (2) a set of constraints that the learner must satisfy. The baseline policy can arise from demonstration data or a teacher agent and may provide useful cues for learning, but it might also be sub-optimal for the task at hand, and is not guaranteed to satisfy the specified constraints, which might encode safety, fairness or other application-specific requirements. In order to safely learn from baseline policies, we propose an iterative policy optimization algorithm that alternates between maximizing expected return on the task, minimizing distance to the baseline policy, and projecting the policy onto the constraint-satisfying set. We analyze our algorithm theoretically and provide a finite-time convergence guarantee. In our experiments on five different control tasks, our algorithm consistently outperforms several state-of-the-art baselines, achieving 10 times fewer constraint violations and 40% higher reward on average.

Motivation & Objective

• To address the challenge of safely learning from baseline policies that may violate safety, fairness, or application-specific constraints.
• To improve sample efficiency and constraint satisfaction in reinforcement learning by leveraging expert or demonstrated policies without requiring them to be constraint-compliant.
• To develop a method that ensures constraint satisfaction at every learning iteration while enabling effective exploration and adaptation.
• To provide a finite-time convergence guarantee for the proposed algorithm under realistic assumptions.
• To empirically validate the method on diverse tasks, including robotic control and traffic management, with real-world applicability in mind.

Proposed method

• The algorithm performs three iterative steps: (1) trust region policy optimization to maximize expected return, (2) projection in policy space to control distance from the baseline policy, and (3) projection onto the feasible constraint set to ensure safety.
• The distance to the baseline policy is adaptively controlled via a dynamic threshold $ h_D^k $, allowing flexible exploration without over-relying on the potentially sub-optimal baseline.
• A trust region approach (e.g., TRPO) ensures stable policy updates and prevents catastrophic policy drift during optimization.
• The constraint projection step enforces feasibility at every iteration, guaranteeing that all policies generated satisfy the given safety or fairness constraints.
• The method avoids relying on cost function learning for the baseline policy, reducing computational overhead and enabling direct use of demonstration data.
• The algorithm is theoretically analyzed and shown to converge in finite time under standard assumptions, with convergence guarantees derived from the interplay of trust region, distance control, and constraint projection.

Experimental results

Research questions

• RQ1 Can a reinforcement learning agent safely learn from a baseline policy that violates desired constraints, while still improving task performance?
• RQ2 How does dynamically adjusting the distance to the baseline policy affect constraint satisfaction and learning efficiency?
• RQ3 Does the proposed method outperform existing safe RL algorithms in terms of both sample efficiency and constraint violation rates?
• RQ4 Can the algorithm generalize across diverse tasks with different types of constraints, such as safety in robotics and fairness in traffic management?
• RQ5 What is the impact of the initial distance threshold $ h_D^0 $ on the final policy performance and constraint compliance?

Key findings

• SPACE achieved 40% higher average return compared to state-of-the-art baselines across five control tasks, including MuJoCo environments and real-world traffic management simulations.
• The algorithm reduced constraint violations by a factor of 10 on average compared to existing methods, demonstrating strong safety guarantees.
• In the car-racing task with human demonstration data, SPACE successfully learned a safe policy even though the baseline human policy violated the cost constraints.
• The initial value of $ h_D^0 $ had minimal impact on performance, indicating robustness to hyperparameter choice.
• Empirical results showed consistent performance gains across five runs with standard deviation reported, confirming reliability and reproducibility.
• The method demonstrated finite-time convergence, providing theoretical assurance for practical deployment.