[Paper Review] Counter-example Guided Learning of Bounds on Environment Behavior

Yuxiao Chen, Sumanth Dathathri+2 more|arXiv (Cornell University)|Jan 20, 2020

Formal Methods in Verification16 references4 citations

TL;DR

This paper proposes a data-driven, counter-example guided framework to learn conservative, state-dependent bounds on unmodeled environment behavior for verifying safety-critical controllers. By combining falsification with support vector machines on real data, it iteratively refines reactive bounds that enable formal verification without requiring an accurate environmental model, successfully verifying a lane-change controller using real human-driving data with 99.9% confidence.

ABSTRACT

There is a growing interest in building autonomous systems that interact with complex environments. The difficulty associated with obtaining an accurate model for such environments poses a challenge to the task of assessing and guaranteeing the system's performance. We present a data-driven solution that allows for a system to be evaluated for specification conformance without an accurate model of the environment. Our approach involves learning a conservative reactive bound of the environment's behavior using data and specification of the system's desired behavior. First, the approach begins by learning a conservative reactive bound on the environment's actions that captures its possible behaviors with high probability. This bound is then used to assist verification, and if the verification fails under this bound, the algorithm returns counter-examples to show how failure occurs and then uses these to refine the bound. We demonstrate the applicability of the approach through two case-studies: i) verifying controllers for a toy multi-robot system, and ii) verifying an instance of human-robot interaction during a lane-change maneuver given real-world human driving data.

Motivation & Objective

• To enable formal verification of autonomous system controllers without requiring an accurate model of complex, uncontrolled environments.
• To address the challenge of verifying safety-critical systems when environment dynamics are unknown or too complex to model explicitly.
• To develop a feedback loop that uses falsifying counter-examples to iteratively refine conservative bounds on environment behavior.
• To provide probabilistic guarantees on the correctness of the learned environment bounds using random convex programming theory.

Proposed method

• The framework uses falsification to generate counter-examples where a controller violates a specification written in Signal Temporal Logic (STL).
• Falsifying traces are treated as negative data, while naturalistic environment behavior data is used as positive data.
• A support vector machine (SVM) is trained on positive and negative data to learn a reactive bound on environment actions that varies with system state.
• The reactive bound is a conservative over-approximation of feasible environment behaviors, ensuring that if the controller is verified safe under this bound, it is safe under the true environment with high probability.
• The process iterates: if falsification succeeds, the bound is refined using new counter-examples; if not, the bound is deemed sufficient for verification.
• Theoretical guarantees on the bound's reliability are derived using Random Convex Programming (RCP), ensuring high-confidence coverage of true environment behavior.

Experimental results

Research questions

• RQ1 Can we verify a controller's safety without modeling the full dynamics of a complex, uncontrolled environment?
• RQ2 How can we learn a conservative, state-dependent bound on environment behavior from real data and system specifications?
• RQ3 Can counter-examples from falsification be used to iteratively refine environment bounds to enable successful verification?
• RQ4 What is the role of the controller's policy and the system specification in guiding the learning of environment behavior bounds?
• RQ5 How can we provide probabilistic guarantees on the correctness of the learned environment bounds?

Key findings

• The framework successfully verified a model predictive control (MPC) controller for a lane-change maneuver using real human-driving data from the SPMD database.
• With the reactive bound learned in 4 iterations, falsification was no longer able to generate a counter-example, confirming that the controller satisfies the STL specification with high confidence.
• The method achieved verification in 29.6 seconds on average for the lane change case, demonstrating computational efficiency.
• When the collision avoidance constraint was removed, falsification successfully identified a collision-causing behavior, providing actionable feedback for controller redesign.
• The reactive bound for the human vehicle's acceleration was learned as an upper bound $ a^{ ext{max}}_{ ext{max}}(x) $, which varied with the state $ x $, capturing the environment's reactive behavior.
• The approach demonstrated robustness on both a toy multi-robot system and a real-world autonomous driving scenario, validating its generalizability.