[Paper Review] NNLander-VeriF: A Neural Network Formal Verification Framework for Vision-Based Autonomous Aircraft Landing

Ulices Santa Cruz, Yasser Shoukry|arXiv (Cornell University)|Mar 29, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

NNLander-VeriF is a formal verification framework that models the camera image formation process geometrically as a ReLU neural network, enabling end-to-end verification of vision-based autonomous aircraft landing systems using off-the-shelf NN model checkers. It achieves formal safety verification of a trained 2-layer ReLU controller across 500 regions in under 76 seconds per region on average.

ABSTRACT

In this paper, we consider the problem of formally verifying a Neural Network (NN) based autonomous landing system. In such a system, a NN controller processes images from a camera to guide the aircraft while approaching the runway. A central challenge for the safety and liveness verification of vision-based closed-loop systems is the lack of mathematical models that captures the relation between the system states (e.g., position of the aircraft) and the images processed by the vision-based NN controller. Another challenge is the limited abilities of state-of-the-art NN model checkers. Such model checkers can reason only about simple input-output robustness properties of neural networks. This limitation creates a gap between the NN model checker abilities and the need to verify a closed-loop system while considering the aircraft dynamics, the perception components, and the NN controller. To this end, this paper presents NNLander-VeriF, a framework to verify vision-based NN controllers used for autonomous landing. NNLander-VeriF addresses the challenges above by exploiting geometric models of perspective cameras to obtain a mathematical model that captures the relation between the aircraft states and the inputs to the NN controller. By converting this model into a NN (with manually assigned weights) and composing it with the NN controller, one can capture the relation between aircraft states and control actions using one augmented NN. Such an augmented NN model leads to a natural encoding of the closed-loop verification into several NN robustness queries, which state-of-the-art NN model checkers can handle. Finally, we evaluate our framework to formally verify the properties of a trained NN and we show its efficiency.

Motivation & Objective

• Address the lack of formal models linking aircraft states to camera images in vision-based autonomous systems.
• Overcome the limitation of state-of-the-art NN model checkers, which can only verify simple input-output robustness properties.
• Enable formal verification of closed-loop systems by integrating aircraft dynamics, perception, and NN control into a single augmented neural network.
• Provide a scalable framework for verifying safety and liveness properties in vision-based autonomous aircraft landing.
• Demonstrate the feasibility and efficiency of formal verification for real-world, high-dimensional perception systems using geometric modeling and off-the-shelf tools.

Proposed method

• Model the perspective camera image formation process using geometric principles, mapping aircraft states (position, orientation) to pixel coordinates.
• Encode the geometric image formation model as a ReLU neural network with manually assigned weights based on camera intrinsic parameters.
• Compose the perception NN with the trained NN controller to form an augmented NN that maps aircraft states directly to control actions.
• Use bounded model checking to generate finite-state machines (FSMs) representing system evolution over time, with safety constraints encoded as Boolean formulas.
• Apply PeregriNN, a state-of-the-art NN model checker, to verify robustness properties of the augmented NN across multiple initial conditions.
• Use SAT4J to implement the finite-state machine verification component, performing bounded model checking with a horizon of 20 steps.

Experimental results

Research questions

• RQ1 Can a geometric model of the camera image formation process be encoded as a ReLU neural network to enable formal verification of vision-based controllers?
• RQ2 To what extent can existing NN model checkers verify complex closed-loop system properties when the perception and control components are composed into a single augmented network?
• RQ3 How efficiently can formal verification be performed on a vision-based autonomous landing system using this framework?
• RQ4 What is the scalability of the framework in terms of image resolution and verification time across different initial conditions?
• RQ5 How does the framework handle safety verification of the NN controller across diverse landing trajectories and unsafe state regions?

Key findings

• The framework successfully verified the safety specification φ = □¬q_unsafe across 500 randomly sampled initial conditions, with the NN controller found to be safe in all cases.
• Average verification time per region was 76 seconds using PeregriNN, demonstrating practical efficiency for real-world deployment.
• Verification time increased monotonically from 2000 seconds (μ=0.1) to 7000 seconds (μ=1.1) due to larger finite-state machine sizes, confirming scalability trade-offs.
• The augmented neural network (NN_aug) enabled end-to-end verification by encoding the full mapping from aircraft states to control actions, making it amenable to existing NN model checkers.
• The use of a piecewise affine (PWA) approximation for image formation allowed exact encoding as a ReLU network, though with potential approximation error.
• The framework demonstrated that formal verification of vision-based systems is feasible when perception is modeled geometrically and composed with the controller.