[Paper Review] Reciprocal Collision Avoidance for General Nonlinear Agents using Reinforcement Learning

Hao Li, Bowen Weng+3 more|arXiv (Cornell University)|Oct 24, 2019

Robotic Path Planning Algorithms19 references4 citations

TL;DR

This paper proposes a reinforcement learning (RL)-based decentralized collision avoidance method for general nonlinear agents with continuous action spaces, combining RL-trained two-agent policies with ORCA-derived safety constraints to ensure collision-free, smooth trajectories. The approach enables fast, scalable multi-agent navigation without real-time communication, outperforming prior RL methods in safety and generalization across complex, dynamic scenarios with nonlinear dynamics.

ABSTRACT

Finding feasible and collision-free paths for multiple nonlinear agents is challenging in the decentralized scenarios due to limited available information of other agents and complex dynamics constraints. In this paper, we propose a fast multi-agent collision avoidance algorithm for general nonlinear agents with continuous action space, where each agent observes only positions and velocities of nearby agents. To reduce online computation, we first decompose the multi-agent scenario and solve a two agents collision avoidance problem using reinforcement learning (RL). When extending the trained policy to a multi-agent problem, safety is ensured by introducing the optimal reciprocal collision avoidance (ORCA) as linear constraints and the overall collision avoidance action could be found through simple convex optimization. Most existing RL-based multi-agent collision avoidance algorithms rely on the direct control of agent velocities. In sharp contrasts, our approach is applicable to general nonlinear agents. Realistic simulations based on nonlinear bicycle agent models are performed with various challenging scenarios, indicating a competitive performance of the proposed method in avoiding collisions, congestion and deadlock with smooth trajectories.

Motivation & Objective

• To address the challenge of decentralized, real-time collision avoidance for general nonlinear agents with limited inter-agent communication.
• To overcome the limitations of existing RL-based methods that assume direct velocity control and lack safety guarantees.
• To enable scalable, safe navigation in complex scenarios involving multiple agents with nonlinear dynamics, such as bicycle models.
• To ensure system-wide safety and avoid deadlock or congestion in symmetric or high-density configurations.
• To combine sample-efficient RL training with convex optimization via ORCA constraints for fast, reliable action selection.

Proposed method

• Train a deep deterministic policy gradient (DDPG) agent on a two-agent collision avoidance task using only relative positions and velocities as observations.
• Decompose the multi-agent problem into pairwise interactions and apply the trained RL policy to each pair to generate candidate avoidance actions.
• Enforce safety through linear velocity constraints derived from the Optimal Reciprocal Collision Avoidance (ORCA) framework, ensuring collision-free motion.
• Formulate the overall multi-agent action as a convex optimization problem that combines individual RL actions while respecting ORCA constraints.
• Use a simple, efficient optimization step at inference time to compute the final action, minimizing online computation.
• Extend the trained policy to multi-agent scenarios with varying numbers of agents, including large-scale setups with up to 42 agents.

Experimental results

Research questions

• RQ1 Can a learned RL policy for two-agent collision avoidance be effectively generalized to multi-agent systems with nonlinear dynamics?
• RQ2 How can safety be systematically guaranteed in a decentralized RL-based multi-agent navigation system without real-time communication?
• RQ3 To what extent does combining RL with ORCA constraints improve safety and performance compared to pure RL or ORCA alone?
• RQ4 Can the proposed method avoid deadlock and congestion in symmetric or high-density agent configurations?
• RQ5 How does the integration of ORCA constraints affect the smoothness and realism of generated trajectories?

Key findings

• The proposed method successfully generated collision-free, smooth trajectories for up to eight agents in symmetric, high-risk configurations, including the classic ORCA deadlock scenario.
• In a 42-agent circular formation, the system achieved safe navigation by having agents turn right and pass the origin from the left, avoiding collisions through ORCA-constrained actions.
• For a 20-agent task involving co-centric circular motion, the method produced safe and coordinated trajectories despite complex, bidirectional interactions between agents.
• Ablation analysis in the 24-agent example showed that ORCA constraints were essential—without them, the RL policy alone would have caused a collision, which was corrected by ORCA's action modification.
• The method achieved fast inference through convex optimization, enabling real-time performance despite complex nonlinear agent dynamics.
• The approach demonstrated strong generalization, handling dynamic agent counts and complex scenarios not seen during training.