[Paper Review] KCRL: Krasovskii-Constrained Reinforcement Learning with Guaranteed Stability in Nonlinear Dynamical Systems

Sahin Lale, Yuanyuan Shi+4 more|arXiv (Cornell University)|Jun 3, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

KCRL proposes a model-based reinforcement learning framework that guarantees asymptotic stability for unknown nonlinear dynamical systems by embedding Krasovskii's Lyapunov function construction as a linear matrix inequality constraint in policy optimization. Using Random Fourier Features to learn the system dynamics and a primal-dual algorithm to enforce stability, KCRL achieves finite-time stabilization with a sample complexity upper bound of $\mathcal{\tilde{O}}\left(\left(\frac{2\bar{G}\|M\|(1+L_{u})+\|M\|(1+L_{u})^{2}}{\bar{\epsilon}-\epsilon_{pd}}\right)^{3}\right)$, ensuring stabilizing policies after $D^2$ samples.

ABSTRACT

Learning a dynamical system requires stabilizing the unknown dynamics to avoid state blow-ups. However, current reinforcement learning (RL) methods lack stabilization guarantees, which limits their applicability for the control of safety-critical systems. We propose a model-based RL framework with formal stability guarantees, Krasovskii Constrained RL (KCRL), that adopts Krasovskii's family of Lyapunov functions as a stability constraint. The proposed method learns the system dynamics up to a confidence interval using feature representation, e.g. Random Fourier Features. It then solves a constrained policy optimization problem with a stability constraint based on Krasovskii's method using a primal-dual approach to recover a stabilizing policy. We show that KCRL is guaranteed to learn a stabilizing policy in a finite number of interactions with the underlying unknown system. We also derive the sample complexity upper bound for stabilization of unknown nonlinear dynamical systems via the KCRL framework.

Motivation & Objective

• To address the lack of formal stability guarantees in model-free RL for safety-critical nonlinear systems.
• To integrate control-theoretic stability analysis into online RL by leveraging Krasovskii’s method for Lyapunov function construction.
• To provide finite-time sample complexity bounds for learning stabilizing policies in unknown nonlinear dynamical systems.
• To ensure stability is preserved despite model uncertainty by constraining the policy optimization via learned Jacobian estimates.

Proposed method

• KCRL uses Random Fourier Features (RFF) to represent and learn the unknown system dynamics within confidence intervals.
• It formulates a constrained policy optimization problem where the stability constraint is derived from Krasovskii’s method, expressed as a linear matrix inequality on the Jacobian of the closed-loop system.
• The framework employs a primal-dual optimization approach to solve the constrained problem, ensuring the stability constraint is satisfied upon convergence.
• The method guarantees that the learned policy stabilizes the true system if the estimated Jacobian remains within a bounded error margin of the true Jacobian.
• It leverages fill distance and approximation error bounds to ensure the stability condition holds over the entire state space.
• The algorithm operates in epochs, with the epoch length set to $D^2$, where $D$ is the number of RFF used in the model representation.

Experimental results

Research questions

• RQ1 Can a model-based RL framework guarantee finite-time stabilization for unknown nonlinear dynamical systems without requiring a stabilizing oracle?
• RQ2 How can Krasovskii’s method for Lyapunov function construction be integrated into a learning-based policy optimization framework to ensure stability?
• RQ3 What sample complexity is required to learn a stabilizing policy using RFF-based dynamics estimation and constrained optimization?
• RQ4 How does model uncertainty affect the stability of the learned policy, and can this be bounded to ensure robustness?

Key findings

• KCRL guarantees asymptotic stability of the closed-loop system around the equilibrium set $S_e$ after $D^2$ samples, provided the model estimate error is bounded.
• The sample complexity upper bound for stabilization is $\mathcal{\tilde{O}}\left(\left(\frac{2\bar{G}\|M\|(1+L_{u})+\|M\|(1+L_{u})^{2}}{\bar{\epsilon}-\epsilon_{pd}}\right)^{3}\right)$, where $\bar{\epsilon}$ is the true stability margin and $\epsilon_{pd}$ is the approximation error bound.
• The primal-dual method ensures that the Krasovskii stability constraint is satisfied at convergence, even with learned dynamics.
• The estimation error $\varepsilon_J$ of the Jacobian is bounded as $\mathcal{\tilde{O}}(D^{-1/3})$ after $D^2$ samples, which ensures stability when $\epsilon_i \leq \bar{\epsilon} - \epsilon_{pd}$.
• The framework ensures stability over the entire state space by requiring the fill distance $h$ to satisfy $\bar{\epsilon} - \epsilon_{pd} > 0$, with $\epsilon_{pd} = 2\bar{G}\|M\|M_G h$.
• KCRL achieves stabilization in a single epoch when the epoch length is set to $D^2$, and subsequent epochs maintain stability due to non-increasing estimation errors.