[Paper Review] Reachability Analysis and Safety Verification for Neural Network Control Systems

Weiming Xiang, Taylor T. Johnson|arXiv (Cornell University)|May 25, 2018

Adversarial Robustness in Machine LearningComputer Science18 references42 citations

TL;DR

This paper develops LP-based methods to over-approximate the output of feedforward neural network controllers and uses them with ODE reachability to verify safety of neural network controlled systems.

ABSTRACT

Autonomous cyber-physical systems (CPS) rely on the correct operation of numerous components, with state-of-the-art methods relying on machine learning (ML) and artificial intelligence (AI) components in various stages of sensing and control. This paper develops methods for estimating the reachable set and verifying safety properties of dynamical systems under control of neural network-based controllers that may be implemented in embedded software. The neural network controllers we consider are feedforward neural networks called multilayer perceptrons (MLP) with general activation functions. As such feedforward networks are memoryless, they may be abstractly represented as mathematical functions, and the reachability analysis of the network amounts to range (image) estimation of this function provided a set of inputs. By discretizing the input set of the MLP into a finite number of hyper-rectangular cells, our approach develops a linear programming (LP) based algorithm for over-approximating the output set of the MLP with its input set as a union of hyper-rectangular cells. Combining the over-approximation for the output set of an MLP based controller and reachable set computation routines for ordinary difference/differential equation (ODE) models, an algorithm is developed to estimate the reachable set of the closed-loop system. Finally, safety verification for neural network control systems can be performed by checking the existence of intersections between the estimated reachable set and unsafe regions. The approach is implemented in a computational software prototype and evaluated on numerical examples.

Motivation & Objective

• Motivate and formalize safety concerns for dynamical systems governed by neural network controllers.
• Develop a practical method to over-approximate the MLP output given bounded inputs.
• Integrate neural network reachability with plant reachable set computations to estimate closed-loop reachability.
• Provide a safety verification procedure by checking intersections with unsafe regions.

Proposed method

• Represent neural networks as mathematical functions via feedforward MLPs.
• Discretize the neural network input space into a union of hyper-rectangles (hyper-rectangular sets).
• Transform output estimation for each hyper-rectangle into a linear programming (LP) problem under activation monotonicity assumptions.
• Propagate the input over-approximation through network layers in a layer-by-layer (MLP) fashion to obtain an output over-approximation.
• Combine the MLP output over-approximation with reachability methods for ODE/difference equations to estimate the closed-loop reachable set over a time horizon.
• Formulate safety verification as checking the non-intersection of the estimated reachable set with unsafe regions.

Experimental results

Research questions

• RQ1 How can the output set of a general-activation-function MLP be over-approximated starting from bounded inputs?
• RQ2 How can an MLP output over-approximation be propagated through multiple layers to obtain a tight overall bound?
• RQ3 How can the neural-network-controlled plant be analyzed by coupling MLP reachability with conventional plant reachability to estimate the closed-loop reachable set?
• RQ4 Under what conditions can safety be verified by checking intersections with unsafe regions using the computed over-approximations?

Key findings

• An LP-based algorithm to over-approximate the output set of an MLP with general activation functions is developed.
• A layer-by-layer approach enables applying the single-layer LP results to multi-layer networks.
• The method produces a computable over-approximation of the closed-loop reachable set by combining MLP reachability with ODE/difference-equation reachability.
• A safety verification procedure is provided that tests for intersections between the estimated reachable set and unsafe regions.
• A computational prototype is implemented and evaluated on numerical examples to demonstrate applicability.