[Paper Review] SCRIMP: Scalable Communication for Reinforcement- and Imitation-Learning-Based Multi-Agent Pathfinding

Yutong Wang, Bairan Xiang+2 more|arXiv (Cornell University)|Mar 1, 2023

Robotic Path Planning Algorithms4 citations

TL;DR

SCRIMP proposes a scalable, communication-augmented reinforcement and imitation learning framework for multi-agent pathfinding (MAPF) that enables agents to learn collision-free paths using only a 3×3 field of view. By leveraging a modified Transformer-based global communication mechanism, stochastic tie-breaking, and intrinsic rewards, SCRIMP achieves performance comparable to centralized planners while maintaining high scalability across large teams.

ABSTRACT

Trading off performance guarantees in favor of scalability, the Multi-Agent Path Finding (MAPF) community has recently started to embrace Multi-Agent Reinforcement Learning (MARL), where agents learn to collaboratively generate individual, collision-free (but often suboptimal) paths. Scalability is usually achieved by assuming a local field of view (FOV) around the agents, helping scale to arbitrary world sizes. However, this assumption significantly limits the amount of information available to the agents, making it difficult for them to enact the type of joint maneuvers needed in denser MAPF tasks. In this paper, we propose SCRIMP, where agents learn individual policies from even very small (down to 3x3) FOVs, by relying on a highly-scalable global/local communication mechanism based on a modified transformer. We further equip agents with a state-value-based tie-breaking strategy to further improve performance in symmetric situations, and introduce intrinsic rewards to encourage exploration while mitigating the long-term credit assignment problem. Empirical evaluations on a set of experiments indicate that SCRIMP can achieve higher performance with improved scalability compared to other state-of-the-art learning-based MAPF planners with larger FOVs, and even yields similar performance as a classical centralized planner in many cases. Ablation studies further validate the effectiveness of our proposed techniques. Finally, we show that our trained model can be directly implemented on real robots for online MAPF through high-fidelity simulations in gazebo.

Motivation & Objective

• To address the scalability and performance limitations of decentralized, partial-observability-based MAPF planners that struggle with complex joint maneuvers.
• To enable agents with minimal local observation (3×3 FOV) to achieve high success rates through scalable, dynamic communication.
• To mitigate the long-term credit assignment problem and improve exploration in sparse-reward environments.
• To design a communication mechanism that avoids the 'chatter problem' while enabling effective team coordination.
• To validate the real-world deployability of the trained policy through high-fidelity robot simulation.

Proposed method

• Introduces a modified Transformer encoder to dynamically fuse messages from other agents, enabling scalable and selective information sharing across the team.
• Employs a state-value-based stochastic tie-breaking mechanism to resolve vertex and swap conflicts by prioritizing agents based on learned long-term team value.
• Incorporates an intrinsic reward mechanism using an episodic buffer to encourage exploration and improve goal-reaching probability.
• Trains agents using a hybrid approach combining deep reinforcement learning and imitation learning to stabilize training and improve sample efficiency.
• Uses a fully decentralized inference setup with only local observations and global communication, enabling real-time online planning.
• Employs a self-message mechanism in the absence of inter-agent communication to isolate the contribution of the communication module.

Experimental results

Research questions

• RQ1 Can a highly scalable, differentiable communication mechanism based on a modified Transformer improve performance in decentralized MAPF with minimal local observation?
• RQ2 Does a learned, state-value-based stochastic tie-breaking strategy outperform random or fixed-priority conflict resolution in symmetric MAPF scenarios?
• RQ3 To what extent do intrinsic rewards based on episodic exploration improve learning efficiency and success rates in sparse-reward MAPF environments?
• RQ4 How does the performance of SCRIMP compare to state-of-the-art learning-based and classical centralized planners across varying team sizes and task complexities?
• RQ5 Can a policy trained in a randomized, unstructured environment be directly deployed on real robots in a structured, high-fidelity simulation without retraining?

Key findings

• SCRIMP achieves success rates comparable to the classical centralized planner ODrM* in most tasks, including complex scenarios requiring joint maneuvers.
• In challenging tasks with high agent density and complex layouts, SCRIMP outperforms state-of-the-art learning-based planners, even those with larger fields of view.
• Ablation studies confirm that removing the communication mechanism (SCRIMP-without-com) leads to the lowest success rate, validating its critical role in complex tasks.
• The stochastic tie-breaking mechanism significantly improves performance over random or fixed-priority strategies, especially in symmetric conflict scenarios.
• The intrinsic reward mechanism increases exploration and reduces the long-term credit assignment problem, leading to higher success rates in complex tasks compared to the baseline with only extrinsic rewards.
• The trained SCRIMP policy was successfully deployed on 16 real robots in Gazebo without retraining, demonstrating real-time, online collision-free path planning in a high-fidelity simulation.