[Paper Review] Reachable Set Estimation for Neural Network Control Systems: A Simulation-Guided Approach

Weiming Xiang, Hoang-Dung Tran+2 more|arXiv (Cornell University)|Apr 26, 2020

Adversarial Robustness in Machine Learning36 references4 citations

TL;DR

This paper proposes a simulation-guided, interval arithmetic-based method for over-approximating the reachable set of feedforward neural networks with general activation functions, enabling efficient safety verification of closed-loop neural network control systems. By integrating simulation-guided output set estimation with ODE reachability analysis, the approach reduces computational cost by 97% compared to prior methods and successfully verifies safety in robotic arm and adaptive cruise control systems.

ABSTRACT

The vulnerability of artificial intelligence (AI) and machine learning (ML) against adversarial disturbances and attacks significantly restricts their applicability in safety-critical systems including cyber-physical systems (CPS) equipped with neural network components at various stages of sensing and control. This paper addresses the reachable set estimation and safety verification problems for dynamical systems embedded with neural network components serving as feedback controllers. The closed-loop system can be abstracted in the form of a continuous-time sampled-data system under the control of a neural network controller. First, a novel reachable set computation method in adaptation to simulations generated out of neural networks is developed. The reachability analysis of a class of feedforward neural networks called multilayer perceptrons (MLP) with general activation functions is performed in the framework of interval arithmetic. Then, in combination with reachability methods developed for various dynamical system classes modeled by ordinary differential equations, a recursive algorithm is developed for over-approximating the reachable set of the closed-loop system. The safety verification for neural network control systems can be performed by examining the emptiness of the intersection between the over-approximation of reachable sets and unsafe sets. The effectiveness of the proposed approach has been validated with evaluations on a robotic arm model and an adaptive cruise control system.

Motivation & Objective

• To address the challenge of safety verification in neural network control systems due to adversarial robustness and black-box nature.
• To develop an efficient, sound over-approximation of the output reachable set for multilayer perceptrons (MLPs) with general activation functions.
• To integrate neural network reachability with continuous-time dynamical system analysis for closed-loop safety verification.
• To reduce computational cost in reachable set estimation while maintaining soundness and accuracy.
• To validate the method on real-world control systems such as robotic arms and adaptive cruise control.

Proposed method

• A simulation-guided approach is developed to over-approximate the output reachable set of feedforward neural networks using interval arithmetic and sensitivity analysis.
• The method formulates output set estimation as four convex optimization problems based on maximal sensitivity, guided by simulations to avoid unnecessary computation.
• For continuous-time dynamics, CORA is used to perform ODE reachability analysis over sampling intervals.
• A recursive algorithm combines neural network output over-approximation with ODE reachability to compute the closed-loop system's reachable set.
• Safety verification is performed by checking the emptiness of the intersection between the over-approximated reachable set and unsafe sets.
• The approach is applied to a sampled-data neural network controller model, integrating simulation guidance with interval-based neural network analysis.

Experimental results

Research questions

• RQ1 Can a simulation-guided method efficiently over-approximate the output reachable set of feedforward neural networks with general activation functions?
• RQ2 How can the computational cost of neural network reachable set estimation be reduced while preserving soundness?
• RQ3 Can the proposed method enable effective safety verification for closed-loop neural network control systems?
• RQ4 To what extent does the method outperform prior simulation-based approaches in terms of efficiency and accuracy?
• RQ5 How well does the method scale and verify safety in real-world control systems like robotic arms and adaptive cruise control?

Key findings

• The proposed simulation-guided method reduces computational cost to just 3% of the prior simulation-based approach [10] for the same interval reachability results in the robotic arm example.
• The method successfully verifies safety in an adaptive cruise control system, showing that the relative distance remains above the safe threshold over a 5-second horizon.
• Reachable set estimations for both lead and ego vehicles in the ACC system are visualized, confirming no intersection with unsafe regions.
• The approach enables sound safety verification by over-approximating the reachable set of the closed-loop system using interval arithmetic and simulation guidance.
• The method is general and applicable to various activation functions, with potential for extension to ReLU networks and other architectures.
• The implementation is publicly available at https://github.com/xiangweiming/ignnv, supporting reproducibility and further research.