[Paper Review] Tracking Progress in Multi-Agent Path Finding

Bojie Shen, Zhe Chen+3 more|arXiv (Cornell University)|May 15, 2023

Multimodal Machine Learning Applications5 citations

TL;DR

This paper introduces a comprehensive visualization and benchmarking platform to track progress in Multi-Agent Path Finding (MAPF), enabling large-scale comparisons of optimal and suboptimal solvers across 1.5 million standardized instances. By visualizing solution quality, congestion patterns, and solver performance, the platform identifies bottlenecks—such as long paths and temporal obstacles—and reveals that current solvers like LNS2 and LaCAM fail on highly congested instances due to computational limits, suggesting opportunities for hybrid approaches.

ABSTRACT

Multi-Agent Path Finding (MAPF) is an important core problem for many new and emerging industrial applications. Many works appear on this topic each year, and a large number of substantial advancements and performance improvements have been reported. Yet measuring overall progress in MAPF is difficult: there are many potential competitors, and the computational burden for comprehensive experimentation is prohibitively large. Moreover, detailed data from past experimentation is usually unavailable. In this work, we introduce a set of methodological and visualisation tools which can help the community establish clear indicators for state-of-the-art MAPF performance and which can facilitate large-scale comparisons between MAPF solvers. Our objectives are to lower the barrier of entry for new researchers and to further promote the study of MAPF, since progress in the area and the main challenges are made much clearer.

Motivation & Objective

• To address the lack of standardized, large-scale evaluation and visibility in MAPF research due to high computational costs and missing experimental data.
• To establish clear, reproducible indicators for state-of-the-art performance in MAPF, including solution quality and progress on unsolved instances.
• To lower the barrier to entry for new researchers by providing accessible, visual, and interactive tools to understand solver behavior and remaining challenges.
• To identify key failure modes in leading solvers—such as LNS2 and LaCAM—through detailed visual analysis of congestion and path planning inefficiencies.
• To promote community-wide progress by enabling public submission, validation, and comparison of solvers on a shared, dynamic platform.

Proposed method

• The authors conduct large-scale experiments on the full 1.5 million-instance MAPF benchmark suite, covering 33 maps from six domains, including game grids, street maps, mazes, rooms, open areas, and warehouses.
• They implement and evaluate multiple leading optimal and suboptimal solvers, including CBSH2-RCT, BCP, ID-CBS, EECBS, FEECBS+, PIBT2, and LNS2, across all instances.
• A custom visualization framework is developed to analyze solver behavior, particularly congestion patterns, waiting queues, and conflict resolution strategies in high-density scenarios.
• The platform tracks lower and upper bounds for each instance, enabling quantification of the optimality gap and attribution of progress to specific algorithms.
• The system supports dynamic updates and future extensions to include wall-clock time measurements via executable submissions on fixed hardware.
• The approach enables hybridization insights, such as combining LaCAM’s fast initial solution with LNS2’s improvement phase, to overcome individual algorithmic limitations.

i

Experimental results

Research questions

• RQ1 How can the MAPF community establish reliable, scalable, and reproducible indicators for state-of-the-art performance across diverse solvers?
• RQ2 What are the primary failure modes of leading MAPF solvers when faced with high-density, congested environments?
• RQ3 Why do some solvers, such as LNS2, fail to solve certain large-scale instances despite strong performance on smaller ones?
• RQ4 Can visual analysis of path planning and conflict resolution reveal systematic inefficiencies that are not captured by standard metrics like success rate or makespan?
• RQ5 How can a shared, dynamic platform improve transparency, reproducibility, and progress tracking in MAPF research?

Key findings

• The LNS2 solver fails on the even-1 instance of the orz900d map with 4,119 agents due to excessive path planning time, caused by long paths and numerous temporal obstacles in congested areas.
• LaCAM exhibits queueing behavior around obstacle corners due to its Push-and-Swap strategy, which prioritizes individual optimality over global coordination, leading to delays.
• The visual analysis reveals that agents in high-congestion zones form queues and wait, indicating that individually optimal paths are suboptimal in collective settings.
• The platform identifies that CBS-based solvers often run out of memory before reaching time limits, highlighting scalability constraints beyond runtime.
• A hybrid approach combining LaCAM’s fast initial solution with LNS2’s improvement phase is shown to be highly effective, addressing the weaknesses of both algorithms.
• The platform enables precise attribution of instance closures to specific algorithms and quantifies the optimality gap, providing a clear benchmark for future progress.

i