[Paper Review] From Model-Based to Data-Driven Simulation: Challenges and Trends in Autonomous Driving

Ferdinand Mütsch, Helen Gremmelmaier+4 more|arXiv (Cornell University)|May 23, 2023

Simulation Techniques and Applications8 citations

TL;DR

A survey of simulation approaches for autonomous driving, highlighting a shift from model-based to data-driven and mixed neural simulations, and outlining content, behavior, and perception realism challenges alongside standardization and validity concerns.

ABSTRACT

Simulation is an integral part in the process of developing autonomous vehicles and advantageous for training, validation, and verification of driving functions. Even though simulations come with a series of benefits compared to real-world experiments, various challenges still prevent virtual testing from entirely replacing physical test-drives. Our work provides an overview of these challenges with regard to different aspects and types of simulation and subsumes current trends to overcome them. We cover aspects around perception-, behavior- and content-realism as well as general hurdles in the domain of simulation. Among others, we observe a trend of data-driven, generative approaches and high-fidelity data synthesis to increasingly replace model-based simulation.

Motivation & Objective

• Clarify a hierarchical classification of autonomous driving simulation approaches and how they differ in realism and methodology.
• Identify current challenges in content, behavior, and perception realism in AV simulations.
• Highlight emerging data-driven and neural simulation trends as solutions to realism gaps.
• Discuss horizontal challenges such as standardization, data/compute needs, and transferability.
• Offer guidance on future research directions and open questions for AV simulation.

Proposed method

• Proposes a hierarchical taxonomy of simulation levels from log replay to mixed neural simulation.
• Reviews literature across content realism, behavior realism, and perception realism to map trends.
• Analyzes challenges and trends associated with each realism dimension.
• Synthesizes data-driven approaches (e.g., NeRFs, diffusion models, GANs) and their impact on content realism.
• Discusses advantages and limitations of data-driven versus model-based simulations and potential for mixed approaches.

Figure 1 : Examples of our proposed simulation levels. Top-left to bottom-right: AR-enhanced (level 1), SUMO (level 2), CARLA (level 3), Block-NeRF (level 4) [ 59 ] .

Experimental results

Research questions

• RQ1 How can simulation approaches for autonomous driving be classified to reflect different realism and methodological levels?
• RQ2 What are the key challenges in achieving content, behavior, and perception realism in AV simulation?
• RQ3 What trends and methods are emerging to address realism gaps, and how do they influence future simulation levels?
• RQ4 What horizontal challenges (standardization, data/compute, validity, transferability) affect the adoption of new simulation paradigms?

Key findings

• A hierarchical scheme of simulation levels (0 to 5) is proposed to compare AV simulators by how they simulate content, dynamics, and sensors.
• Content realism is increasingly supported by data-driven methods (e.g., NeRFs, GANs, diffusion models) and procedurally generated content, reducing the need for hand-crafted assets.
• Behavior realism is moving toward ML-based sequence models, transformers, adversarial RL, and knowledge-guided methods to better capture long-tail and critical scenarios.
• Perception realism benefits from data-driven sensor modeling, few-shot learning, and learned noise models to improve camera, lidar, and radar fidelity.
• Horizontal challenges include standardization (OpenSCENARIO/OpenDRIVE/OpenOSI), data/compute requirements, and assessing validity and transferability of simulations.
• Overall, the field is converging toward higher levels of data-driven and mixed neural simulations, with several open research questions remaining.

Figure 2 : Our proposed hierarchy of simulation levels