[Paper Review] Safety-Critical Modular Deep Reinforcement Learning with Temporal Logic through Gaussian Processes and Control Barrier Functions

Mingyu Cai, Cristian-Ioan Vasile|arXiv (Cornell University)|Sep 7, 2021

Advanced Control Systems Optimization55 references4 citations

TL;DR

This paper proposes a safety-critical modular deep reinforcement learning framework that integrates linear temporal logic (LTL) specifications, Gaussian processes for uncertainty modeling, and exponential control barrier functions (ECBFs) to ensure safe exploration and high-probability satisfaction of complex tasks in continuous state and action spaces. The method achieves near-perfect success rates with strong safety guarantees during training in robotic environments.

ABSTRACT

Reinforcement learning (RL) is a promising approach and has limited success towards real-world applications, because ensuring safe exploration or facilitating adequate exploitation is a challenges for controlling robotic systems with unknown models and measurement uncertainties. Such a learning problem becomes even more intractable for complex tasks over continuous space (state-space and action-space). In this paper, we propose a learning-based control framework consisting of several aspects: (1) linear temporal logic (LTL) is leveraged to facilitate complex tasks over an infinite horizons which can be translated to a novel automaton structure; (2) we propose an innovative reward scheme for RL-agent with the formal guarantee such that global optimal policies maximize the probability of satisfying the LTL specifications; (3) based on a reward shaping technique, we develop a modular policy-gradient architecture utilizing the benefits of automaton structures to decompose overall tasks and facilitate the performance of learned controllers; (4) by incorporating Gaussian Processes (GPs) to estimate the uncertain dynamic systems, we synthesize a model-based safeguard using Exponential Control Barrier Functions (ECBFs) to address problems with high-order relative degrees. In addition, we utilize the properties of LTL automatons and ECBFs to construct a guiding process to further improve the efficiency of exploration. Finally, we demonstrate the effectiveness of the framework via several robotic environments. And we show such an ECBF-based modular deep RL algorithm achieves near-perfect success rates and guard safety with a high probability confidence during training.

Motivation & Objective

• To address the challenge of safe exploration and exploitation in deep reinforcement learning for robotic systems with unknown dynamics and measurement uncertainties.
• To enable complex, long-horizon tasks over continuous spaces by formalizing them using linear temporal logic (LTL) specifications.
• To develop a reward shaping mechanism that guarantees global optimality in satisfying LTL specifications through a novel automaton-based structure.
• To integrate model-based safety via Gaussian processes and exponential control barrier functions (ECBFs), particularly for systems with high-order relative degrees.
• To improve sample efficiency and policy performance through a modular policy-gradient architecture guided by LTL automaton structure and ECBF constraints.

Proposed method

• Leverages linear temporal logic (LTL) to encode complex, infinite-horizon tasks and translates them into a novel automaton structure for policy learning.
• Designs a reward shaping scheme that formally guarantees that the optimal policy maximizes the probability of satisfying the LTL specifications.
• Develops a modular policy-gradient architecture that decomposes the overall task using the automaton structure to improve learning efficiency and controller performance.
• Employs Gaussian processes (GPs) to model uncertain system dynamics and estimate model uncertainty in real time.
• Introduces exponential control barrier functions (ECBFs) to synthesize a model-based safety layer that ensures constraint satisfaction even for systems with high-order relative degrees.
• Constructs a guiding process using LTL automaton properties and ECBF constraints to accelerate safe exploration and improve sample efficiency.

Experimental results

Research questions

• RQ1 How can LTL specifications be effectively encoded and leveraged to guide deep reinforcement learning in continuous state and action spaces?
• RQ2 Can a reward shaping mechanism be designed to formally guarantee that the optimal policy maximizes the probability of satisfying LTL specifications?
• RQ3 How can modular policy-gradient architectures improve learning performance and scalability in complex robotic control tasks?
• RQ4 In what way can Gaussian processes and exponential control barrier functions jointly ensure safety for systems with high-order relative degrees?
• RQ5 To what extent does the integration of LTL automaton structure and ECBF constraints improve the efficiency and safety of exploration in deep RL?

Key findings

• The proposed framework achieves near-perfect success rates in multiple robotic control environments, demonstrating high reliability in task completion.
• The integration of ECBFs with Gaussian processes ensures safe exploration and constraint satisfaction with high probability confidence during training.
• The modular policy-gradient architecture significantly improves learning efficiency and performance compared to non-modular baselines.
• The reward shaping mechanism formally guarantees that the global optimal policy maximizes the probability of satisfying the LTL specifications.
• The guiding process based on LTL automaton and ECBF properties accelerates convergence and enhances sample efficiency.
• The framework successfully handles systems with high-order relative degrees, a challenge often unaddressed in prior safe RL methods.