[Paper Review] Provably Correct Training of Neural Network Controllers Using Reachability Analysis

Xiaowu Sun, Yasser Shoukry|arXiv (Cornell University)|Feb 22, 2021

Adversarial Robustness in Machine Learning46 references4 citations

TL;DR

This paper proposes a framework that trains ReLU neural network controllers with provable safety and liveness guarantees by combining reachability analysis with a novel weight projection operator. It constructs a finite-state abstract model of the system to identify safe and liveness-satisfying Continuous Piece-Wise Affine (CPWA) functions, which are then enforced during training via projection, ensuring correctness without post-hoc verification or runtime monitoring.

ABSTRACT

In this paper, we consider the problem of training neural network (NN) controllers for nonlinear dynamical systems that are guaranteed to satisfy safety and liveness (e.g., reach-avoid) properties. Our approach is to combine model-based design methodologies for dynamical systems with data-driven approaches to achieve this target. We confine our attention to NNs with Rectifier Linear Unit (ReLU) nonlinearity which are known to represent Continuous Piece-Wise Affine (CPWA) functions. Given a mathematical model of the dynamical system, we compute a finite-state abstract model that captures the closed-loop behavior under all possible CPWA controllers. Using this finite-state abstract model, our framework identifies a family of CPWA functions guaranteed to satisfy the safety requirements. We augment the learning algorithm with a NN weight projection operator during training that enforces the resulting NN to represent a CPWA function from the provably safe family of CPWA functions. Moreover, the proposed framework uses the finite-state abstract model to identify candidate CPWA functions that may satisfy the liveness properties. Using such candidate CPWA functions, the proposed framework biases the NN training to achieve the liveness specification. We show the efficacy of the proposed framework both in simulation and on an actual robotic vehicle.

Motivation & Objective

• To address the lack of formal safety and reliability guarantees in data-driven neural network controllers for safety-critical cyber-physical systems.
• To develop a training framework that ensures neural network controllers satisfy both safety and liveness properties without relying on post-training verification or runtime monitoring.
• To integrate model-based reachability analysis with data-driven training to guide the learning process toward provably correct controllers.
• To enable real-world deployment of neural network controllers on robotic platforms with formal correctness guarantees.

Proposed method

• Construct a finite-state abstract model of the nonlinear dynamical system that captures the closed-loop behavior under all possible Continuous Piece-Wise Affine (CPWA) controllers.
• Use reachability analysis to compute reachable sets and identify a family of CPWA functions that guarantee safety under all initial conditions.
• Introduce a novel NN weight projection operator during training that constrains the learned weights to represent only CPWA functions from the provably safe family.
• Leverage the abstract model to identify candidate CPWA functions that may satisfy liveness properties, rank them, and bias the training process accordingly.
• Apply the framework to a robotic vehicle (PiCar) using Proximal Policy Optimization (PPO) with the projection operator to train a controller that satisfies safety and liveness.
• Enable parallelization of key components such as reachable set computation, posterior graph construction, and local neural network training for scalability.

Experimental results

Research questions

• RQ1 Can a data-driven training framework for neural network controllers guarantee safety and liveness without post-hoc verification or runtime monitoring?
• RQ2 How can reachability analysis be used to abstract the behavior of all possible ReLU neural network controllers and identify a safe family of CPWA functions?
• RQ3 To what extent can a weight projection operator enforce the learned controller to remain within a provably safe set of CPWA functions during training?
• RQ4 How can the abstract model be used to guide training toward liveness satisfaction, given that liveness cannot be enforced via projection alone?
• RQ5 What is the scalability of the framework with respect to system dimension and partition granularity in real-world robotic control tasks?

Key findings

• The framework successfully trained a neural network controller for a PiCar robotic vehicle that satisfied the safety property φ_safety over multiple laps on a race track, as demonstrated in simulation and real-world deployment.
• Execution time for the framework grows linearly with the number of abstract states and controller partitions, indicating good scalability for moderate partitioning.
• For increasing system dimension, the number of abstract states and execution time grow exponentially, but the framework still handles 10-dimensional systems in under 10,000 seconds (≈2.7 hours) for the posterior graph construction.
• The framework achieves provably correct controllers without requiring online monitoring, predictive filters, barrier functions, or post-training formal verification.
• The weight projection operator effectively constrains the neural network to represent only CPWA functions from the safe family, ensuring safety by construction.
• The use of abstract model-based candidate ranking and training biasing enables convergence to controllers that satisfy liveness, even though liveness cannot be enforced via projection alone.