[Paper Review] AirSim Drone Racing Lab

Ratnesh Madaan, Nicholas Gydé+8 more|arXiv (Cornell University)|Dec 14, 2019

Robotic Path Planning Algorithms33 references14 citations

TL;DR

This paper introduces AirSim Drone Racing Lab, a high-fidelity simulation framework for autonomous drone racing that enables rapid prototyping of perception, planning, and control algorithms using photo-realistic environments, multiple sensor modalities (including monocular, depth, optical flow, and neuromorphic events), and a comprehensive API. The framework was used to host a simulation-based competition at NeurIPS 2019, demonstrating its utility for machine learning research in drone autonomy with strong sim-to-real transfer potential.

ABSTRACT

Autonomous drone racing is a challenging research problem at the intersection of computer vision, planning, state estimation, and control. We introduce AirSim Drone Racing Lab, a simulation framework for enabling fast prototyping of algorithms for autonomy and enabling machine learning research in this domain, with the goal of reducing the time, money, and risks associated with field robotics. Our framework enables generation of racing tracks in multiple photo-realistic environments, orchestration of drone races, comes with a suite of gate assets, allows for multiple sensor modalities (monocular, depth, neuromorphic events, optical flow), different camera models, and benchmarking of planning, control, computer vision, and learning-based algorithms. We used our framework to host a simulation based drone racing competition at NeurIPS 2019. The competition binaries are available at our github repository.

Motivation & Objective

• Address the high barrier to entry in autonomous drone racing research due to cost, risk, and complexity of real-world experimentation.
• Provide a simulation environment that closely mimics real-world conditions to enable robust testing and generalization of perception, planning, and control algorithms.
• Facilitate machine learning research by offering a scalable, accessible, and extensible platform for training and benchmarking end-to-end autonomy systems.
• Bridge the sim-to-real gap through photo-realistic rendering, domain randomization, and support for diverse sensor modalities including event cameras and optical flow.
• Host a large-scale, simulation-only competition (Game of Drones at NeurIPS 2019) to engage the machine learning community in drone racing challenges.

Proposed method

• Built on top of AirSim and Unreal Engine to leverage high-fidelity graphics and physics simulation for photo-realistic environments and accurate drone dynamics.
• Provided a comprehensive API for low-level control (e.g., angle rate and thrust commands), medium-level control (velocity and position setpoints), and high-level trajectory planning (spline fitting and trajectory tracking).
• Enabled generation of racing tracks with configurable complexity, including variable curvature and gate visibility metrics for evaluating planning and perception challenges.
• Integrated support for multiple sensor modalities: monocular RGB, depth, optical flow, and neuromorphic event-based sensing via GPU-accelerated compute shaders.
• Implemented domain randomization via texture randomization and camera parameter variation to improve sim-to-real generalization.
• Enabled real-time event generation using GPU-based shaders that compute pixel intensity differences and convert them into asynchronous event data.

Experimental results

Research questions

• RQ1 Can a high-fidelity simulation framework reduce the time, cost, and risk of developing autonomous drone racing systems while maintaining strong sim-to-real transfer capabilities?
• RQ2 How effective is the integration of diverse sensor modalities—especially neuromorphic events and optical flow—within a simulation environment for training end-to-end learning-based agents?
• RQ3 To what extent can a simulation-based competition attract and engage the machine learning community in solving complex, multi-module autonomy problems in drone racing?
• RQ4 How do track complexity metrics such as curvature and gate visibility correlate with the difficulty of perception, planning, and control tasks in autonomous drone racing?
• RQ5 Can the use of domain randomization and realistic sensor modeling in simulation improve the generalization of learned policies to real-world drone racing scenarios?

Key findings

• The AirSim Drone Racing Lab framework successfully enabled the organization of a large-scale, simulation-only drone racing competition at NeurIPS 2019, attracting diverse participation from the machine learning community.
• The framework supports high-fidelity simulation with photo-realistic rendering, enabling effective training and evaluation of vision-based and learning-based algorithms.
• Real-time generation of neuromorphic event data and optical flow using GPU-accelerated shaders allows for low-latency, high-performance simulation of asynchronous and motion-based sensing.
• Track complexity metrics—curvature and gate visibility—were quantitatively visualized and used to assess the difficulty of planning and perception tasks across different race environments.
• The inclusion of multiple sensor modalities, including monocular, depth, optical flow, and event-based sensing, enabled comprehensive benchmarking of perception and control algorithms.
• The framework’s extensible API and support for domain randomization facilitated robust policy training and improved sim-to-real generalization for autonomous drone racing.