[Paper Review] Automatic Testing and Falsification with Dynamically Constrained Reinforcement Learning

Xin Qin, Nikos Aréchiga+2 more|arXiv (Cornell University)|Oct 30, 2019

Formal Methods in Verification16 citations

TL;DR

This paper proposes an agent-centric reinforcement learning framework that integrates dynamically constrained rules—expressed in Signal Temporal Logic (STL)—to train adversarial agents for automatic testing and falsification of cyber-physical systems like autonomous vehicles. The method enables modular, reusable agents that find all dangerous behaviors detectable by traditional falsification while respecting hierarchical traffic rules, demonstrated on two automotive case studies.

ABSTRACT

We consider the problem of using reinforcement learning to train adversarial agents for automatic testing and falsification of cyberphysical systems, such as autonomous vehicles, robots, and airplanes. In order to produce useful agents, however, it is useful to be able to control the degree of adversariality by specifying rules that an agent must follow. For example, when testing an autonomous vehicle, it is useful to find maximally antagonistic traffic participants that obey traffic rules. We model dynamic constraints as hierarchically ordered rules expressed in Signal Temporal Logic, and show how these can be incorporated into an agent training process. We prove that our agent-centric approach is able to find all dangerous behaviors that can be found by traditional falsification techniques while producing modular and reusable agents. We demonstrate our approach on two case studies from the automotive domain.

Motivation & Objective

• To develop a reinforcement learning-based approach for automatic testing and falsification of cyber-physical systems, such as autonomous vehicles and aircraft.
• To enable control over adversarial behavior by encoding dynamic constraints as hierarchically ordered rules in Signal Temporal Logic (STL).
• To ensure that trained agents discover all dangerous system behaviors identifiable by traditional falsification techniques.
• To produce modular and reusable agents that respect realistic operational constraints, such as traffic rules.
• To validate the approach on real-world automotive case studies involving complex traffic scenarios.

Proposed method

• Model dynamic constraints as hierarchically ordered rules in Signal Temporal Logic (STL), enabling structured specification of valid behaviors.
• Integrate STL constraints into the reinforcement learning training process to guide agent exploration within feasible and realistic behavior spaces.
• Use an agent-centric training paradigm where each agent is trained to maximize adversariality while adhering to specified STL rules.
• Formulate the reward function to penalize constraint violations and encourage discovery of system failures under realistic, rule-abiding adversarial conditions.
• Apply hierarchical rule enforcement to prioritize critical safety constraints during training, ensuring compliance with traffic regulations.
• Train agents end-to-end using deep reinforcement learning, with policy optimization guided by STL-based reward shaping and constraint enforcement.

Experimental results

Research questions

• RQ1 Can reinforcement learning agents be trained to discover dangerous behaviors in cyber-physical systems while respecting realistic, rule-based constraints?
• RQ2 How does incorporating hierarchical Signal Temporal Logic (STL) constraints affect the effectiveness and safety of adversarial testing agents?
• RQ3 Can the proposed agent-centric approach achieve the same coverage of failure modes as traditional falsification techniques?
• RQ4 To what extent are the trained agents modular and reusable across different testing scenarios?
• RQ5 How do dynamically constrained agents compare to unconstrained adversarial agents in terms of realism and test coverage?

Key findings

• The proposed method successfully discovers all dangerous behaviors that are detectable by traditional falsification techniques, ensuring full coverage.
• Agents trained with STL constraints produce realistic, rule-compliant adversarial behaviors that simulate plausible traffic scenarios.
• The hierarchical enforcement of constraints ensures that critical safety rules are never violated during agent training.
• The agent-centric design enables modularity and reusability across different test scenarios and system configurations.
• The approach achieves effective test coverage while maintaining operational realism, as demonstrated in two automotive case studies.
• The integration of STL constraints into the RL training loop enables scalable and verifiable adversarial testing of cyber-physical systems.