[Paper Review] QuadSwarm: A Modular Multi-Quadrotor Simulator for Deep Reinforcement Learning with Direct Thrust Control

Zhehui Huang, Sumeet Batra+8 more|arXiv (Cornell University)|Jun 15, 2023

Reinforcement Learning in Robotics4 citations

TL;DR

QuadSwarm is a high-speed, modular, Python-based simulator for deep reinforcement learning (RL) in single and multi-quadrotor systems, featuring direct per-rotor thrust control, physics-based dynamics for Crazyflie 2.x, and domain randomization to enable zero-shot sim2real transfer. It achieves over 62,000 simulation samples per second on 16 CPU cores with eight quadrotors, enabling fast, scalable, and parallelizable training for complex multi-agent flight policies.

ABSTRACT

Reinforcement learning (RL) has shown promise in creating robust policies for robotics tasks. However, contemporary RL algorithms are data-hungry, often requiring billions of environment transitions to train successful policies. This necessitates the use of fast and highly-parallelizable simulators. In addition to speed, such simulators need to model the physics of the robots and their interaction with the environment to a level acceptable for transferring policies learned in simulation to reality. We present QuadSwarm, a fast, reliable simulator for research in single and multi-robot RL for quadrotors that addresses both issues. QuadSwarm, with fast forward-dynamics propagation decoupled from rendering, is designed to be highly parallelizable such that throughput scales linearly with additional compute. It provides multiple components tailored toward multi-robot RL, including diverse training scenarios, and provides domain randomization to facilitate the development and sim2real transfer of multi-quadrotor control policies. Initial experiments suggest that QuadSwarm achieves over 48,500 simulation samples per second (SPS) on a single quadrotor and over 62,000 SPS on eight quadrotors on a 16-core CPU. The code can be found in https://github.com/Zhehui-Huang/quad-swarm-rl.

Motivation & Objective

• To address the need for fast, parallelizable, and physically accurate simulators for training deep RL policies in multi-quadrotor systems.
• To enable robust sim2real transfer of quadrotor control policies by incorporating realistic physics and domain randomization.
• To support diverse, unified training scenarios and interaction-based rewards for multi-agent coordination.
• To provide a fully Python-based, modular simulator that simplifies integration with deep RL libraries and accelerates research iteration.

Proposed method

• Decouples physics simulation from rendering using Numba-optimized Python code to enable high-speed, single-threaded throughput.
• Implements an ad hoc physics engine with direct per-rotor thrust control and accurate dynamics modeling for the Crazyflie 2.x platform.
• Uses domain randomization across physical parameters (e.g., mass, drag, thrust) to improve policy generalization and sim2real transfer.
• Supports multiple training scenarios, including goal-reaching, formation flight, and collision avoidance, with a unified reward function.
• Integrates with the Sample Factory RL library to accelerate training via synchronous/asynchronous PPO algorithms.
• Employs a modular design with reusable components for quadrotor dynamics, collision detection, and reward shaping.

Experimental results

Research questions

• RQ1 Can a fully Python-based simulator achieve high simulation throughput while maintaining accurate physics for multi-quadrotor RL?
• RQ2 How does direct per-rotor thrust control improve policy learning and sim2real transfer in multi-agent quadrotor systems?
• RQ3 To what extent does domain randomization in the simulator enhance the generalization and robustness of trained policies?
• RQ4 Can a modular simulator with diverse training scenarios and interaction-based rewards improve multi-agent coordination in complex environments?
• RQ5 How does QuadSwarm’s performance scale with increasing numbers of quadrotors and compute resources?

Key findings

• QuadSwarm achieves over 48,500 simulation samples per second (SPS) on a single quadrotor and over 62,000 SPS on eight quadrotors using a 16-core CPU.
• The simulator maintains high performance even with collision detection, demonstrating efficient parallelization and scalability.
• Policies trained in QuadSwarm successfully transfer to real hardware in both single- and multi-quadrotor scenarios with zero-shot deployment.
• The integration with Sample Factory enables fast wall-clock training, reducing time-to-trial for complex multi-agent policies.
• The inclusion of diverse training scenarios and interaction-based rewards leads to more robust and generalizable control policies.
• QuadSwarm outperforms existing simulators like gym-pybullet-drones by 2x in speed and offers superior support for multi-agent RL and per-rotor control.