[Paper Review] Cosys-AirSim: A Real-Time Simulation Framework Expanded for Complex Industrial Applications

Wouter Jansen, Erik Verreycken+5 more|arXiv (Cornell University)|Mar 23, 2023

Robotics and Sensor-Based Localization4 citations

TL;DR

This paper presents Cosys-AirSim, an open-source, real-time simulation framework extended from AirSim to support complex industrial applications through enhanced sensor modalities, procedural environment generation, and support for autonomous navigation and transfer learning. The framework enables high-fidelity, physically accurate simulation of LiDAR, radar, and vision sensors in dynamic, procedurally generated environments, significantly reducing the sim-to-real gap for robotics and AI training in industrial settings.

ABSTRACT

Within academia and industry, there has been a need for expansive simulation frameworks that include model-based simulation of sensors, mobile vehicles, and the environment around them. To this end, the modular, real-time, and open-source AirSim framework has been a popular community-built system that fulfills some of those needs. However, the framework required adding systems to serve some complex industrial applications, including designing and testing new sensor modalities, Simultaneous Localization And Mapping (SLAM), autonomous navigation algorithms, and transfer learning with machine learning models. In this work, we discuss the modification and additions to our open-source version of the AirSim simulation framework, including new sensor modalities, vehicle types, and methods to generate realistic environments with changeable objects procedurally. Furthermore, we show the various applications and use cases the framework can serve.

Motivation & Objective

• To address the growing need for flexible, real-time simulation frameworks that support complex industrial robotics and AI applications beyond standard ADAS/AD use cases.
• To extend the open-source AirSim framework with new sensor modalities, including LiDDR, pulse-echo radar, and multi-spectral sensors, for realistic data generation.
• To enable the procedural generation of complex, changeable indoor and outdoor environments with customizable objects and structural constraints.
• To support the development and validation of autonomous navigation, SLAM, and transfer learning pipelines using synthetic data from accurate sensor simulations.
• To bridge the sim-to-real gap by ensuring high-fidelity sensor and vehicle modeling for real-world deployment of robotic systems.

Proposed method

• Extended the AirSim framework with new sensor plugins for LiDAR, radar, and multi-spectral imaging, enabling physically accurate simulation of sensor data.
• Implemented procedural generation of indoor and outdoor environments using rule-based constraints and object placement algorithms to create diverse, realistic, and dynamic scenes.
• Integrated real-time simulation of ground and aerial vehicles (e.g., AGVs, drones) with accurate kinematics and dynamics for autonomous navigation testing.
• Enabled the generation of large-scale, labeled datasets for LiDAR point clouds and depth images to train deep learning models for semantic segmentation.
• Provided a simulation-to-real transfer pipeline by aligning simulated sensor data with real-world experiments, validated through SLAM and navigation benchmarking.
• Supported integration with real-time tracking systems to record, replay, and augment training data for ML models.

Figure 1: Block diagram of the software connection between the Unreal Engine and the AirSim Framework and their sub-elements. Furthermore, the major extensions discussed in this paper are listed on the right side.

Experimental results

Research questions

• RQ1 Can a real-time, open-source simulation framework be extended to support complex industrial sensor modalities such as LiDAR and pulse-echo radar with high physical fidelity?
• RQ2 To what extent can procedural generation of environments with dynamic, rule-based object placement improve the realism and diversity of synthetic training data?
• RQ3 How accurately can simulation-based training and validation of SLAM and navigation algorithms replicate real-world performance in industrial environments?
• RQ4 What is the magnitude of the sim-to-real gap when using synthetic data from the extended framework for training deep learning models in robotics applications?
• RQ5 Can the framework support efficient, scalable data generation for UAVs in no-fly or high-risk zones where real-world data collection is restricted?

Key findings

• The Cosys-AirSim framework achieved a navigation deviation of ±1.9% between simulated and real-world experiments for a differential drive AGV with pulse-echo radar.
• The SLAM algorithm in simulation had an error of less than 3.2 m in 95% of the trajectory, significantly outperforming the real-world odometry-based algorithm, which had a minimum error of 10 m in 85% of the trajectory.
• The framework enabled the generation of physically accurate, labeled LiDAR datasets for indoor environments, which are otherwise scarce and costly to collect manually.
• Procedural environment generation allowed rapid creation of diverse, realistic indoor and outdoor scenes tailored to specific industrial use cases, such as warehouses and greenhouses.
• The simulation framework successfully reduced the sim-to-real gap for radar-based navigation and SLAM, validating its use for real-world algorithm development and testing.
• The open-source extension supports scalable data generation for UAVs, enabling training of segmentation models in synthetic outdoor environments where real-world data collection is restricted.

Figure 2: ( a ) Differential drive steering model vehicles like the Clearpath Robotics Husky and Pioneer P3-DX were added. ( b ) The unique object-based instance segmentation gives each object a unique label and color.