[Paper Review] Black-Box Safety Validation of Autonomous Systems: A Multi-Fidelity Reinforcement Learning Approach

Jared Beard, Ali Baheri|arXiv (Cornell University)|Mar 7, 2022

Software Reliability and Analysis Research4 citations

TL;DR

This paper proposes a multi-fidelity reinforcement learning approach, MF-RL-falsify, that enhances safety validation of autonomous systems by integrating model knowledge—via the 'knows what it knows' (KWIK) framework—into decision-making. By enabling bidirectional knowledge transfer between low- and high-fidelity simulators and using knowledge-based exploration, the method achieves up to 6× higher failure discovery efficiency per sample compared to single-fidelity methods, with minimal high-fidelity sampling and improved exploration in early stages.

ABSTRACT

The increasing use of autonomous and semi-autonomous agents in society has made it crucial to validate their safety. However, the complex scenarios in which they are used may make formal verification impossible. To address this challenge, simulation-based safety validation is employed to test the complex system. Recent approaches using reinforcement learning are prone to excessive exploitation of known failures and a lack of coverage in the space of failures. To address this limitation, a type of Markov decision process called the "knowledge MDP" has been defined. This approach takes into account both the learned model and its metadata, such as sample counts, in estimating the system's knowledge through the "knows what it knows" framework. A novel algorithm that extends bidirectional learning to multiple fidelities of simulators has been developed to solve the safety validation problem. The effectiveness of this approach is demonstrated through a case study in which an adversary is trained to intercept a test model in a grid-world environment. Monte Carlo trials compare the sample efficiency of the proposed algorithm to learning with a single-fidelity simulator and show the importance of incorporating knowledge about learned models into the decision-making process.

Motivation & Objective

• To address the limitations of single-fidelity reinforcement learning in safety validation, where rare failures are often missed and high-fidelity simulations are computationally expensive.
• To improve sample efficiency in failure discovery by leveraging multiple simulator fidelities in a bidirectional learning framework.
• To integrate model knowledge—such as sample counts and uncertainty—into the decision-making process using the 'knows what it knows' (KWIK) framework.
• To reduce reliance on high-fidelity simulations while maximizing failure coverage through knowledge-guided exploration.
• To develop a scalable, knowledge-augmented reinforcement learning algorithm that outperforms standard single-fidelity approaches in early and comprehensive failure detection.

Proposed method

• Introduces a novel Knowledge Markov Decision Process (KMDP) that explicitly models both the system dynamics and the agent’s knowledge about its model, including metadata like sample counts.
• Employs a bidirectional learning mechanism that transfers knowledge both from low- to high-fidelity simulators and vice versa, improving sample efficiency.
• Uses the 'knows what it knows' (KWIK) framework to estimate model uncertainty and guide exploration, preventing premature convergence.
• Applies a marginal update mechanism based on the knowledge gain per sample to dynamically adjust exploration, especially effective when model estimates are under-sampled.
• Leverages value iteration to solve the KMDP and optimize the adversarial policy for failure discovery, with a focus on minimizing high-fidelity simulation usage.
• Employs Monte Carlo trials to compare the multi-fidelity approach (MF-RL-falsify) against single-fidelity baselines across multiple metrics, including failure discovery rate and sample efficiency.

Experimental results

Research questions

• RQ1 How does the integration of model knowledge (via the KWIK framework) improve exploration and failure discovery in multi-fidelity safety validation?
• RQ2 What is the impact of bidirectional knowledge transfer between low- and high-fidelity simulators on sample efficiency and failure coverage?
• RQ3 How does the marginal update mechanism affect exploration, particularly in early and converged phases of learning?
• RQ4 To what extent does the MF-RL-falsify algorithm reduce the number of required high-fidelity samples while maintaining or improving failure detection performance?
• RQ5 How does the multi-fidelity approach compare to single-fidelity reinforcement learning in terms of failure discovery per sample, especially in the initial learning stages?

Key findings

• The MF-RL-falsify algorithm achieves up to 6 times higher failure discovery efficiency per sample in early training stages compared to single-fidelity baselines.
• The ratio of high-fidelity to low-fidelity samples required for convergence is less than 0.4, indicating significant reduction in high-fidelity simulation usage.
• The marginal update with R_inc = 5 increases failure discovery by up to 15% with minimal high-fidelity sampling, demonstrating enhanced exploration in under-sampled regions.
• The multi-fidelity approach discovers failures much earlier than the single-fidelity method, with the latter wasting early high-fidelity samples on unproductive exploration.
• After convergence, both methods exhibit similar sample accumulation rates, but the MF approach maintains a significant advantage in failure discovery per sample throughout the learning process.
• The inclusion of knowledge-based updates prevents early convergence and sustains exploration, particularly beneficial in complex or rare failure scenarios.