[Paper Review] How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligence

Xuemin Hu, Shen Li+4 more|arXiv (Cornell University)|May 2, 2023

Autonomous Vehicle Technology and Safety4 citations

TL;DR

This survey comprehensively reviews sim2real, digital twins (DTs), and parallel intelligence (PI) as key enablers for bridging the reality gap in autonomous driving. It synthesizes state-of-the-art methods—such as domain randomization, meta-learning, and multi-scale simulation—demonstrating how simulation-driven knowledge transfer enhances safety, reduces real-world testing costs, and enables scalable, robust autonomous systems through integrated virtual and physical system co-evolution.

ABSTRACT

Safety and cost are two important concerns for the development of autonomous driving technologies. From the academic research to commercial applications of autonomous driving vehicles, sufficient simulation and real world testing are required. In general, a large scale of testing in simulation environment is conducted and then the learned driving knowledge is transferred to the real world, so how to adapt driving knowledge learned in simulation to reality becomes a critical issue. However, the virtual simulation world differs from the real world in many aspects such as lighting, textures, vehicle dynamics, and agents' behaviors, etc., which makes it difficult to bridge the gap between the virtual and real worlds. This gap is commonly referred to as the reality gap (RG). In recent years, researchers have explored various approaches to address the reality gap issue, which can be broadly classified into three categories: transferring knowledge from simulation to reality (sim2real), learning in digital twins (DTs), and learning by parallel intelligence (PI) technologies. In this paper, we consider the solutions through the sim2real, DTs, and PI technologies, and review important applications and innovations in the field of autonomous driving. Meanwhile, we show the state-of-the-arts from the views of algorithms, models, and simulators, and elaborate the development process from sim2real to DTs and PI. The presentation also illustrates the far-reaching effects and challenges in the development of sim2real, DTs, and PI in autonomous driving.

Motivation & Objective

• Address the critical challenge of the reality gap (RG) between simulated and real-world autonomous driving environments.
• Systematically review and categorize existing methods in sim2real, digital twins (DTs), and parallel intelligence (PI) for autonomous driving.
• Identify key limitations and open challenges in knowledge transfer, data scarcity, and model generalization across simulation and reality.
• Provide a unified framework for evaluating and integrating sim2real, DT, and PI technologies to improve safety, scalability, and real-world deployment of autonomous systems.

Proposed method

• Categorize sim2real techniques into six core methods: curriculum learning, meta-learning, knowledge distillation, robust reinforcement learning, domain randomization, and transfer learning.
• Analyze digital twin (DT) architectures that enable bidirectional data flow between physical vehicles and virtual simulations using real sensor data and physics-based modeling.
• Examine parallel intelligence (PI) frameworks that support description, prediction, and prescriptive control through synchronized computation across physical and virtual systems.
• Survey major autonomous driving simulators (e.g., AirSim, CarSim, SUMO) to evaluate their fidelity, scalability, and suitability for sim2real and DT applications.
• Integrate multi-modal, high-volume data from both real and simulated environments to train and validate PI-based autonomous driving systems.
• Propose a hierarchical evaluation framework to assess the generalization and transferability of learned policies across diverse environments and tasks.

Experimental results

Research questions

• RQ1 How do sim2real techniques such as domain randomization and robust reinforcement learning effectively reduce the reality gap in autonomous driving?
• RQ2 To what extent can digital twins enable accurate real-time modeling and control of physical vehicles through synchronized data and model updates?
• RQ3 What are the key architectural and algorithmic components that enable effective parallel intelligence in autonomous driving systems?
• RQ4 What are the primary limitations in current sim2real, DT, and PI methods that hinder generalization across diverse driving scenarios and environments?
• RQ5 How can the integration of sim2real, DT, and PI technologies lead to more scalable, safe, and cost-effective autonomous vehicle development?

Key findings

• Domain randomization and robust reinforcement learning significantly improve policy generalization by exposing agents to diverse environmental variations during training.
• Digital twins enable real-time motion planning and system monitoring by synchronizing physical vehicle states with high-fidelity virtual replicas using real sensor data.
• Parallel intelligence frameworks enhance system performance by enabling multi-unit, concurrent processing of physical and simulated data streams.
• Current sim2real methods are highly scenario-specific, lacking a general, task-independent approach to knowledge transfer, which limits scalability and portability.
• The scarcity of real-world data—especially in rare or extreme scenarios—remains a major bottleneck, despite the abundance of simulated data.
• A lack of standardized methodologies for modeling, evaluating, and integrating DT and PI components hinders the development of comprehensive, end-to-end autonomous driving systems.