[Paper Review] A Survey on Scenario-Based Testing for Automated Driving Systems in High-Fidelity Simulation

Ziyuan Zhong, Yun Tang+4 more|arXiv (Cornell University)|Dec 2, 2021

Software Testing and Debugging Techniques116 references40 citations

TL;DR

This survey introduces a generic framework for scenario-based testing in high-fidelity simulators for ADS, reviews recent works, and discusses open challenges and future directions. It compares methods by simulators, systems, objectives, scenario spaces, and search algorithms.

ABSTRACT

Automated Driving Systems (ADSs) have seen rapid progress in recent years. To ensure the safety and reliability of these systems, extensive testings are being conducted before their future mass deployment. Testing the system on the road is the closest to real-world and desirable approach, but it is incredibly costly. Also, it is infeasible to cover rare corner cases using such real-world testing. Thus, a popular alternative is to evaluate an ADS's performance in some well-designed challenging scenarios, a.k.a. scenario-based testing. High-fidelity simulators have been widely used in this setting to maximize flexibility and convenience in testing what-if scenarios. Although many works have been proposed offering diverse frameworks/methods for testing specific systems, the comparisons and connections among these works are still missing. To bridge this gap, in this work, we provide a generic formulation of scenario-based testing in high-fidelity simulation and conduct a literature review on the existing works. We further compare them and present the open challenges as well as potential future research directions.

Motivation & Objective

• Propose a generic formulation for scenario-based testing in high-fidelity simulation.
• Review and compare recent works (mainly from 2018 onward) across simulators, target systems, testing objectives, scenario parameters, and search algorithms.
• Identify common challenges and gaps to guide future research in ADS scenario-based testing.
• Discuss open questions and directions to improve fidelity, transferability, and efficiency of testing frameworks.

Proposed method

• Present a generic workflow for scenario-based ADS testing: scenario generation, execution, and evaluation.
• Define a layered scenario representation (functional, logical, concrete) and a scenario layer model to organize parameters.
• Compare works along four dimensions: simulator, system under test, testing objective, and search algorithm.
• Summarize the selection and evaluation criteria used to curate and analyze the literature.
• Discuss a gap analysis between simulation results and real-world performance and propose potential mitigations.

Experimental results

Research questions

• RQ1 What constitutes a generic, reusable framework for scenario-based testing in high-fidelity ADS simulators?
• RQ2 How do existing works differ in terms of simulators, systems under test, testing objectives, scenario parameter spaces, and search strategies?
• RQ3 What are the main challenges in aligning simulation-based testing with real-world safety guarantees, and what directions may address them?

Key findings

• A unified three-component workflow (scenario generation, execution, evaluation) is common across studies.
• A layered (functional, logical, concrete) and five-layer scenario model helps organize scenario parameters for comparison.
• Most works use software-in-loop high-fidelity simulators to test system-level ADS safety in a variety of maps, sensors, and NPC behaviors.
• There is a notable gap in transferring testing results from simulation to real-world deployments, highlighting fidelity and transferability challenges.
• Open challenges include simulator realism, open-source system availability, and generalization of testing methods across platforms.