[Paper Review] Testing of Autonomous Driving Systems: Where Are We and Where Should We Go?

Guannan Lou, Yao Deng+3 more|arXiv (Cornell University)|Jun 23, 2021

Software Testing and Debugging Techniques69 references4 citations

TL;DR

This paper presents a comprehensive empirical study of autonomous driving system (ADS) testing practices and needs through interviews with 10 ADS developers and a survey of 100 practitioners. It identifies seven common testing practices and four emerging needs—corner case detection, test acceleration, scenario authoring, and data labeling—while revealing a significant gap in existing SE research, which largely overlooks multi-module systems and under-investigates practical needs like test speed and tooling support.

ABSTRACT

Autonomous driving has shown great potential to reform modern transportation. Yet its reliability and safety have drawn a lot of attention and concerns. Compared with traditional software systems, autonomous driving systems (ADSs) often use deep neural networks in tandem with logic-based modules. This new paradigm poses unique challenges for software testing. Despite the recent development of new ADS testing techniques, it is not clear to what extent those techniques have addressed the needs of ADS practitioners. To fill this gap, we present the first comprehensive study to identify the current practices and needs of ADS testing. We conducted semi-structured interviews with developers from 10 autonomous driving companies and surveyed 100 developers who have worked on autonomous driving systems. A systematic analysis of the interview and survey data revealed 7 common practices and 4 emerging needs of autonomous driving testing. Through a comprehensive literature review, we developed a taxonomy of existing ADS testing techniques and analyzed the gap between ADS research and practitioners' needs. Finally, we proposed several future directions for SE researchers, such as developing test reduction techniques to accelerate simulation-based ADS testing.

Motivation & Objective

• To understand current industrial practices in testing autonomous driving systems (ADS), particularly in real-world development environments.
• To identify emerging testing needs of ADS practitioners that are not adequately addressed by existing software engineering (SE) research.
• To assess the alignment between existing SE research on ADS testing and the actual needs of industrial practitioners.
• To bridge the gap between academic research and industrial practice by proposing future research directions grounded in empirical evidence.

Proposed method

• Conducted semi-structured interviews with 10 developers from diverse autonomous driving companies to explore real-world testing practices and challenges.
• Designed and distributed a large-scale survey to 1,978 ADS developers, collecting 100 valid responses to quantify and validate interview findings.
• Performed a systematic literature review of 117 SE conference and journal papers, filtering to 42 focused on ADS testing to develop a taxonomy of existing techniques.
• Triangulated qualitative interview data, quantitative survey responses, and literature review results to identify patterns, gaps, and research opportunities.
• Used thematic analysis with inter-rater reliability (Cohen’s Kappa = 0.85) to ensure consistency in coding interview and survey responses.
• Proposed future research directions based on identified gaps, particularly in test reduction, scenario generation, and tooling support for complex ADS workflows.

Experimental results

Research questions

• RQ1 What are the current industrial practices in testing autonomous driving systems?
• RQ2 What are the emerging needs of ADS practitioners for testing, particularly in terms of tooling and methodology?
• RQ3 To what extent do existing software engineering research techniques for ADS testing align with industrial needs and practices?
• RQ4 How do testing practices differ between end-to-end models and multi-module ADS architectures in practice?

Key findings

• Seven common testing practices were identified, including system-level metrics like consistency and latency, which are not typically captured by standard model accuracy metrics.
• Four key emerging needs were revealed: identifying corner cases and unexpected driving scenarios, accelerating testing (especially simulation), tool support for constructing complex driving scenarios, and support for data labeling.
• While corner case detection is well-represented in SE research, test acceleration and scenario/tooling support remain under-investigated in existing literature.
• Most existing ADS testing techniques are designed for end-to-end deep learning models, but industrial systems increasingly rely on modular architectures combining perception networks with logic-based controllers.
• There is a significant misalignment between current SE research and industrial needs, particularly in practical tooling and performance optimization for simulation-based testing.
• The study identifies test reduction techniques as a critical future direction to accelerate simulation testing, which is a major bottleneck in industrial ADS development.