[Paper Review] Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guarantees

Kai–Chieh Hsu, Allen Z. Ren+3 more|arXiv (Cornell University)|Jan 20, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes Sim-to-Lab-to-Real, a framework for safe reinforcement learning that combines Hamilton-Jacobi reachability analysis with PAC-Bayes generalization guarantees to ensure safety and performance in real-world deployment. It uses a dual-policy setup with a performance policy and a safety-aware backup policy, enabling safe exploration in simulation and lab environments, and provides probabilistic certificates of safety and generalization for unseen real environments.

ABSTRACT

Safety is a critical component of autonomous systems and remains a challenge for learning-based policies to be utilized in the real world. In particular, policies learned using reinforcement learning often fail to generalize to novel environments due to unsafe behavior. In this paper, we propose Sim-to-Lab-to-Real to bridge the reality gap with a probabilistically guaranteed safety-aware policy distribution. To improve safety, we apply a dual policy setup where a performance policy is trained using the cumulative task reward and a backup (safety) policy is trained by solving the Safety Bellman Equation based on Hamilton-Jacobi (HJ) reachability analysis. In Sim-to-Lab transfer, we apply a supervisory control scheme to shield unsafe actions during exploration; in Lab-to-Real transfer, we leverage the Probably Approximately Correct (PAC)-Bayes framework to provide lower bounds on the expected performance and safety of policies in unseen environments. Additionally, inheriting from the HJ reachability analysis, the bound accounts for the expectation over the worst-case safety in each environment. We empirically study the proposed framework for ego-vision navigation in two types of indoor environments with varying degrees of photorealism. We also demonstrate strong generalization performance through hardware experiments in real indoor spaces with a quadrupedal robot. See https://sites.google.com/princeton.edu/sim-to-lab-to-real for supplementary material.

Motivation & Objective

• Address the reality gap in sim-to-real reinforcement learning by introducing a controlled intermediate 'Lab' training stage.
• Ensure safety during training and deployment by integrating a backup safety policy derived from HJ reachability analysis.
• Provide probabilistic, distribution-free guarantees on policy performance and safety in unseen real environments using PAC-Bayes theory.
• Bridge the gap between simulation and real-world deployment by certifying generalization through a combination of shielding and statistical bounds.
• Enable reliable deployment of RL policies in safety-critical environments such as homes with children or dynamic indoor spaces.

Proposed method

• Train a dual-policy system in simulation: a performance policy optimized for task reward and a backup safety policy trained via the Safety Bellman Equation using HJ reachability analysis.
• Apply a supervisory control scheme (shielding) in the Lab stage to override unsafe actions from the performance policy when safety critic values exceed a threshold.
• Leverage the PAC-Bayes framework to derive lower bounds on expected performance and safety, accounting for worst-case environment variations.
• Condition the policy on a latent variable distribution to enable generalization across diverse environments during Sim-to-Lab transfer.
• Fine-tune the policy distribution in Lab using realistic simulated environments that mirror real-world photorealism and dynamics.
• Use system identification to model real robot dynamics and camera poses, ensuring fidelity in Lab training and generalization to real deployment.

Experimental results

Research questions

• RQ1 Can a dual-policy framework with shielding ensure safe exploration during Sim-to-Lab training without compromising performance?
• RQ2 How can PAC-Bayes generalization bounds be applied to provide probabilistic guarantees on safety and performance in unseen real environments?
• RQ3 To what extent does the framework reduce safety violations during real-world deployment compared to standard sim-to-real approaches?
• RQ4 How does the choice of latent variable distribution and diversity affect generalization performance and safety robustness?
• RQ5 Can the framework provide certifiable safety and performance guarantees even under small distribution shifts between Lab and Real environments?

Key findings

• The framework significantly reduces safety violations during training through effective shielding by the backup policy, even in high-diversity environments.
• With sufficient latent diversity (e.g., β=2), the policy distribution generalizes well to unseen testing environments while maintaining safety.
• Hardware experiments with a quadrupedal robot in real indoor spaces demonstrate strong generalization and safety performance, validating theoretical guarantees.
• The PAC-Bayes bounds provide tight, probabilistically guaranteed lower bounds on expected performance and safety, even under worst-case environment expectations.
• The method achieves robustness to environment distribution shifts, with theoretical guarantees holding under realistic assumptions of environment modeling.
• The framework enables deployment of RL policies in safety-critical real-world settings with certified performance and safety, supported by empirical validation.