[Paper Review] A Learnable Safety Measure

Steve Heim, Alexander von Rohr+2 more|arXiv (Cornell University)|Oct 7, 2019

Gaussian Processes and Bayesian Inference26 references4 citations

TL;DR

This paper proposes a learnable safety measure that implicitly captures system dynamics relative to failure states, enabling model-free safe exploration via Gaussian process active sampling. The method learns a probabilistic estimate of viable state-action pairs, reducing failures by 8% after 500 samples without requiring accurate system models or prior knowledge of failure boundaries.

ABSTRACT

Failures are challenging for learning to control physical systems since they risk damage, time-consuming resets, and often provide little gradient information. Adding safety constraints to exploration typically requires a lot of prior knowledge and domain expertise. We present a safety measure which implicitly captures how the system dynamics relate to a set of failure states. Not only can this measure be used as a safety function, but also to directly compute the set of safe state-action pairs. Further, we show a model-free approach to learn this measure by active sampling using Gaussian processes. While safety can only be guaranteed after learning the safety measure, we show that failures can already be greatly reduced by using the estimated measure during learning.

Motivation & Objective

• To develop a safety measure that implicitly encodes system dynamics relative to failure states without requiring explicit viability kernel computation.
• To enable safe exploration in physical systems where failures are costly but not critical, especially when accurate dynamics models are unavailable.
• To learn a probabilistic estimate of the viable state-action set using active sampling with Gaussian processes, minimizing reliance on prior knowledge.
• To reduce failure rates during learning by using the estimated safety measure as a guide, even before convergence.
• To provide a scalable, model-free alternative to traditional viability kernel computation that avoids high engineering effort and model inaccuracies.

Proposed method

• The safety measure is defined as the integral of viability over the set of viable state-action pairs, quantifying the 'room to maneuver' without entering failure states.
• A Gaussian process is used to model the safety measure, with a prior informed by low-fidelity simulations and a known operating point to improve convergence.
• Active sampling selects new state-action pairs based on uncertainty and potential to improve the safety estimate, using confidence bounds γ_opt and γ_caut.
• The algorithm iteratively refines the safety measure and identifies the viable set by avoiding unviable states while respecting the smoothness assumption of the GP.
• The method uses a state-action space formulation of viability, where a state-action pair is viable if the system can avoid failure indefinitely from that point.
• The approach balances exploration and safety by prioritizing samples that reduce uncertainty in high-risk regions while minimizing failure occurrences.

Experimental results

Research questions

• RQ1 Can a safety measure be learned directly from data without requiring an accurate system dynamics model?
• RQ2 How can a model-free approach using Gaussian processes enable safe exploration in systems with costly failures?
• RQ3 To what extent can an estimated safety measure reduce failure rates during learning before convergence?
• RQ4 How do tuning parameters like γ_opt, γ_caut, and λ affect the accuracy and conservatism of the learned viable set?
• RQ5 In what ways does the smoothness assumption of the GP impact learning performance near non-smooth failure boundaries?

Key findings

• The proposed safety measure effectively captures the system's viability by quantifying the number of safe actions available from each state-action pair.
• After 500 samples, the method achieved an 8% failure rate in the SLIP model, demonstrating significant reduction in failures during learning.
• The algorithm successfully learned the viable set even near non-smooth failure boundaries, though these regions required more samples due to violation of the smoothness assumption.
• The prior covariance function, derived from a low-fidelity simulation, enabled faster convergence despite model inaccuracies, especially when combined with a well-chosen operating point.
• The method achieved a conservative yet nearly maximal approximation of the viable set, showing robustness to prior inaccuracies and effective use of active sampling.
• The approach enables safe learning in complex, high-dimensional systems where traditional model-based viability kernel computation is infeasible.