[Paper Review] MRPB 1.0: A Unified Benchmark for the Evaluation of Mobile Robot Local Planning Approaches

Jian Wen, Xuebo Zhang+5 more|arXiv (Cornell University)|Nov 1, 2020

Robotic Path Planning Algorithms22 references4 citations

TL;DR

MRPB 1.0 is a unified benchmark for evaluating mobile robot local planning approaches, featuring four simulation scenarios—indoor, narrow space, partially unknown, and dynamic environments—alongside three principled metrics: safety, efficiency, and smoothness. The benchmark demonstrates that optimization-based planners like TEB outperform sampling-based ones like DWA in efficiency (8.94× faster), safety (greater obstacle clearance), and flexibility (superior performance in complex mazes).

ABSTRACT

Local planning is one of the key technologies for mobile robots to achieve full autonomy and has been widely investigated. To evaluate mobile robot local planning approaches in a unified and comprehensive way, a mobile robot local planning benchmark called MRPB 1.0 is newly proposed in this paper. The benchmark facilitates both motion planning researchers who want to compare the performance of a new local planner relative to many other state-of-the-art approaches as well as end users in the mobile robotics industry who want to select a local planner that performs best on some problems of interest. We elaborately design various simulation scenarios to challenge the applicability of local planners, including large-scale, partially unknown, and dynamic complex environments. Furthermore, three types of principled evaluation metrics are carefully designed to quantitatively evaluate the performance of local planners, wherein the safety, efficiency, and smoothness of motions are comprehensively considered. We present the application of the proposed benchmark in two popular open-source local planners to show the practicality of the benchmark. In addition, some insights and guidelines about the design and selection of local planners are also provided. The benchmark website contains all data of the designed simulation scenarios, detailed descriptions of these scenarios, and example code.

Motivation & Objective

• To address the lack of a standardized, repeatable, and comprehensive evaluation framework for mobile robot local planning approaches.
• To provide public, reproducible simulation scenarios that challenge the robustness of local planners in complex, real-world-like environments.
• To establish a principled, multi-dimensional evaluation system using safety, efficiency, and smoothness metrics for objective performance comparison.
• To enable fair and systematic benchmarking of local planners across diverse environments and performance dimensions.
• To guide researchers and practitioners in selecting or designing local planners based on empirical, data-driven insights.

Proposed method

• Design and release four distinct simulation scenario types in Gazebo: indoor (various-scale office-like), narrow space (mazes, U/Z-shaped corridors), partially unknown (incomplete prior maps), and dynamic (moving obstacles).
• Implement a unified evaluation pipeline using open-source ROS-based simulation, with standardized start/goal pose configurations available on the benchmark website.
• Define three core evaluation metrics: safety (minimum distance to obstacles, time spent in danger zones), efficiency (travel distance, time to goal), and smoothness (path curvature, velocity/acceleration changes).
• Apply the benchmark to evaluate two open-source local planners—DWA (sampling-based) and TEB (optimization-based)—across all scenarios to ensure practical validation.
• Use multiple random start-goal pairs per scenario to reduce evaluation randomness and improve statistical reliability.
• Integrate computation efficiency as a metric, acknowledging platform dependency, and report results in absolute time and relative speedup.

Experimental results

Research questions

• RQ1 How can a unified, reproducible, and comprehensive benchmark be designed to evaluate mobile robot local planning approaches across diverse environments?
• RQ2 To what extent do different local planners (e.g., DWA vs. TEB) differ in safety, efficiency, and smoothness under standardized conditions?
• RQ3 How do local planners perform in challenging environments such as narrow passages, partially unknown maps, and dynamic obstacle settings?
• RQ4 What are the key performance trade-offs between sampling-based and optimization-based local planning methods?
• RQ5 Can a principled multi-metric evaluation system reveal insights that single-metric evaluations (e.g., total distance) miss?

Key findings

• TEB achieved 8.94× faster computation than DWA due to its optimization-based approach using the global path as an initial guess.
• TEB improved motion efficiency by 9.2% on average compared to DWA, as measured by reduced time-to-goal across scenarios.
• TEB demonstrated superior safety performance, maintaining a minimum distance to obstacles that was consistently higher than DWA’s, reducing collision risk.
• In narrow space and maze-like environments, DWA frequently failed due to infeasible velocity sampling, while TEB successfully navigated all such scenarios.
• TEB produced smoother trajectories with lower curvature and velocity fluctuations, as quantified by path smoothness metrics.
• The benchmark revealed that DWA’s performance is highly sensitive to hand-tuned cost function weights, highlighting the need for adaptive or learned parameter tuning.