[Paper Review] Constrained Reinforcement Learning for Robotics via Scenario-Based Programming

Davide Corsi, Raz Yerushalmi+4 more|arXiv (Cornell University)|Jun 20, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes a novel constrained reinforcement learning framework that integrates subject-matter-expert knowledge into deep reinforcement learning via scenario-based programming (SBP), enabling safe, high-performance robotic policies. By encoding safety and behavioral constraints as human-readable scenarios and combining them with Lagrangian-PPO, the method achieves significant reductions in rule violations and improved success rates in mapless navigation tasks, with formal verification confirming safety properties.

ABSTRACT

Deep reinforcement learning (DRL) has achieved groundbreaking successes in a wide variety of robotic applications. A natural consequence is the adoption of this paradigm for safety-critical tasks, where human safety and expensive hardware can be involved. In this context, it is crucial to optimize the performance of DRL-based agents while providing guarantees about their behavior. This paper presents a novel technique for incorporating domain-expert knowledge into a constrained DRL training loop. Our technique exploits the scenario-based programming paradigm, which is designed to allow specifying such knowledge in a simple and intuitive way. We validated our method on the popular robotic mapless navigation problem, in simulation, and on the actual platform. Our experiments demonstrate that using our approach to leverage expert knowledge dramatically improves the safety and the performance of the agent.

Motivation & Objective

• To address the lack of intuitive, human-readable methods for encoding safety and behavioral constraints in deep reinforcement learning for robotics.
• To improve the reliability and safety of DRL-based robotic policies in safety-critical environments such as autonomous navigation.
• To integrate expert knowledge directly into the training loop using scenario-based programming (SBP) for better interpretability and control.
• To enable formal verification of safety properties in trained policies, ensuring deployability in real-world systems.
• To demonstrate the effectiveness of the approach on a real robotic platform (Turtlebot3) in both simulation and real-world mapless navigation tasks.

Proposed method

• The method extends Lagrangian-PPO with scenario-based programming (SBP) to encode safety and behavioral constraints as human-readable scenarios.
• Each scenario defines a desired or forbidden behavior (e.g., avoiding back-and-forth rotation, maintaining forward progress) and is translated into a cost function for constrained DRL.
• The SBP framework allows multiple constraints to be composed and executed in parallel, ensuring cohesive system behavior.
• The approach uses a Lagrangian multiplier method to balance the main reward objective and constraint costs, with automatic adaptation of multipliers.
• The method supports formal verification of safety properties using deep neural network verification techniques to validate trained policies.
• The framework is evaluated on a mapless navigation task using the Robotis Turtlebot3 platform in simulation and on hardware.

Experimental results

Research questions

• RQ1 Can scenario-based programming be effectively used to encode complex safety and behavioral constraints in a human-readable way for DRL training?
• RQ2 Does integrating SBP with constrained DRL improve the safety and performance of robotic agents compared to standard DRL or penalty-based methods?
• RQ3 Can the proposed method generate policies that are formally verifiable for safety, even in complex, real-world environments?
• RQ4 How does the method scale with increasing numbers of constraints, and what are the trade-offs in training stability and performance?
• RQ5 To what extent can expert knowledge be injected into the DRL loop without degrading the primary task performance?

Key findings

• The proposed method achieved a 100% success rate in mapless navigation in simulation and on the real Turtlebot3 platform, significantly outperforming standard Lagrangian-PPO and prior penalty-based approaches.
• The method reduced violations of safety rules—such as back-and-forth rotation—by over 90% compared to the baseline method of Yerushalmi et al. [64], which required manual penalty tuning.
• Formal verification confirmed that all trained policies satisfied predefined safety properties, with 100% of tested models being formally safe under the given constraints.
• The approach demonstrated robustness across different environments and constraint sets, maintaining high performance even when multiple constraints were introduced.
• The method eliminated the need for manual penalty tuning, which is a major limitation in existing constrained DRL approaches, by embedding constraints directly into the SBP framework.
• The integration of SBP with Lagrangian-PPO enabled scalable and interpretable constraint specification, making it feasible to incrementally add new behavioral rules.