[Paper Review] Safe Reinforcement Learning via Probabilistic Shields

Nils Jansen, Bettina Könighofer+3 more|arXiv (Cornell University)|Jul 16, 2018

Software Reliability and Analysis Research57 references16 citations

TL;DR

This paper proposes probabilistic shields for reinforcement learning in Markov decision processes (MDPs) to ensure safety with high probability by blocking actions that exceed a predefined risk threshold δ. Using model checking on a safety-relevant MDP fragment, the shield pre-evaluates action risks and guides exploration, enabling faster learning with orders-of-magnitude fewer episodes and significantly fewer safety violations compared to unshielded RL.

ABSTRACT

This paper targets the efficient construction of a safety shield for decision making in scenarios that incorporate uncertainty. Markov decision processes (MDPs) are prominent models to capture such planning problems. Reinforcement learning (RL) is a machine learning technique to determine near-optimal policies in MDPs that may be unknown prior to exploring the model. However, during exploration, RL is prone to induce behavior that is undesirable or not allowed in safety- or mission-critical contexts. We introduce the concept of a probabilistic shield that enables decision-making to adhere to safety constraints with high probability. In a separation of concerns, we employ formal verification to efficiently compute the probabilities of critical decisions within a safety-relevant fragment of the MDP. We use these results to realize a shield that is applied to an RL algorithm which then optimizes the actual performance objective. We discuss tradeoffs between sufficient progress in exploration of the environment and ensuring safety. In our experiments, we demonstrate on the arcade game PAC-MAN and on a case study involving service robots that the learning efficiency increases as the learning needs orders of magnitude fewer episodes.

Motivation & Objective

• To address the challenge of unsafe exploration in reinforcement learning (RL) for safety-critical systems.
• To enable efficient learning under probabilistic safety constraints rather than deterministic ones, allowing controlled risk.
• To decouple safety verification from policy learning via formal model checking on a reduced safety-relevant MDP.
• To balance exploration efficiency and safety by dynamically adjusting the risk threshold δ.
• To demonstrate that shielded RL achieves superior performance and safety with significantly fewer training episodes.

Proposed method

• Construct a safety-relevant MDP fragment from the full MDP using model-based RL to capture only critical states and transitions.
• Use probabilistic model checking (via Storm) to compute the exact probability of reaching unsafe states within a finite horizon (e.g., 10 steps) for each action.
• Define a δ-shield that blocks any action whose safety probability exceeds a threshold δ, with adaptive δ values per state.
• Apply the shield at inference time to filter actions from a Q-learning agent, allowing only safe or low-risk actions.
• Optimize shield construction via piecewise, independent model checking and multi-threading to handle large MDPs.
• Use fallback behavior: if no safe action exists under δ, the shield allows the optimal action to prevent policy collapse.

Experimental results

Research questions

• RQ1 Can probabilistic shields based on model checking improve learning efficiency in RL while ensuring high-probability safety?
• RQ2 How does the choice of δ affect the trade-off between safety and exploration efficiency?
• RQ3 To what extent can shielding reduce the number of episodes required to learn effective policies in safety-critical environments?
• RQ4 How scalable is the shield construction process when applied to large MDPs, especially when using abstraction and parallelization?
• RQ5 Can shielded RL outperform unshielded RL in terms of both performance and safety in real-world scenarios like PAC-MAN and warehouse robotics?

Key findings

• In PAC-MAN, shielded RL achieved significantly higher average scores (e.g., 339.89 vs. -129.25 for 27x25 grid with 4 ghosts) and improved win rates (0.00 vs. 0.00, but 0.84 vs. 0.04 in smaller instances).
• For the warehouse robot case study, shielding 8 decision states increased the win rate from 0.16 to 0.71 and raised the average score from -186 to 420.
• The shield reduced the number of episodes needed to learn effective policies by orders of magnitude, as fewer episodes ended in failure due to safety violations.
• Shield construction for MDPs with up to ~10^6 states took up to 6 hours (single-threaded), but was scalable via piecewise and parallel construction.
• The shield effectively prevented fatal actions (e.g., ghost collisions) even when the optimal policy would have led to high risk, improving robustness and learning stability.
• The adaptive δ mechanism allowed dynamic adjustment of safety permissiveness, enabling exploration while maintaining probabilistic safety guarantees.