[Paper Review] Obstacle-aware Waypoint Generation for Long-range Guidance of Deep-Reinforcement-Learning-based Navigation Approaches

Linh Kästner, Xinlin Zhao+2 more|arXiv (Cornell University)|Sep 23, 2021

Robotic Path Planning Algorithms24 references4 citations

TL;DR

This paper proposes a hierarchical navigation system that uses obstacle-aware waypoint generation via timed Delaunay triangulation and an extended hybrid A*-search in state-time space to guide DRL-based local planners in long-range, dynamic environments. The method achieves 100% success rate and significantly lower collision rates (11–10 collisions) at 0.3–0.5 m/s obstacle speeds, outperforming baseline subgoal methods in safety, robustness, and efficiency.

ABSTRACT

Navigation of mobile robots within crowded environments is an essential task in various use cases, such as delivery, health care, or logistics. Deep Reinforcement Learning (DRL) emerged as an alternative method to replace overly conservative approaches and promises more efficient and flexible navigation. However, Deep Reinforcement Learning is limited to local navigation due to its myopic nature. Previous research works proposed various ways to combine Deep Reinforcement Learning with conventional methods but a common problem is the complexity of highly dynamic environments due to the unpredictability of humans and other objects within the environment. In this paper, we propose a hierarchical waypoint generator, which considers moving obstacles and thus generates safer and more robust waypoints for Deep-Reinforcement-Learning-based local planners. Therefore, we utilize Delaunay Triangulation to encode obstacles and incorporate an extended hybrid A-Star approach to efficiently search for an optimal solution in the time-state space. We compared our waypoint generator against two baseline approaches and outperform them in terms of safety, efficiency, and robustness.

Motivation & Objective

• To address the myopic nature of DRL-based local planners in long-range navigation by providing robust, dynamic global guidance.
• To improve safety and efficiency in highly dynamic environments with moving obstacles, where static global paths become invalid quickly.
• To reduce reliance on frequent global replanning by generating adaptive, obstacle-informed subgoals that remain valid over time.
• To integrate model-based planning with DRL local planning through a hierarchical framework that balances global awareness and local adaptability.
• To evaluate the proposed waypoint generator against baseline methods in terms of success rate, collision count, and computational efficiency.

Proposed method

• Uses timed Delaunay triangulation to encode moving obstacles in the state-time space, enabling efficient spatial-temporal obstacle representation.
• Employs an extended hybrid A*-search algorithm to compute near-optimal trajectories in the time-state space, considering both position and velocity of dynamic obstacles.
• Applies an ESDF-free gradient-based trajectory optimization (EGO) to refine the initial path, ensuring smooth and collision-free motion.
• Generates sparse landmark waypoints as subgoals for the DRL local planner, derived from the optimized trajectory to guide long-range navigation.
• Integrates the mid-level planner with a DRL-based local planner, where waypoints serve as short-term goals to mitigate the myopic limitation of DRL.
• Uses a landmark-based waypoint selection strategy to reduce the computational cost of global replanning, improving scalability in large environments.

Experimental results

Research questions

• RQ1 Can obstacle-aware waypoint generation improve the safety and robustness of DRL-based navigation in highly dynamic environments?
• RQ2 How does considering moving obstacles in the state-time space affect the success rate and collision rate of long-range navigation?
• RQ3 To what extent does the proposed method reduce reliance on frequent global replanning compared to baseline approaches?
• RQ4 How does the performance of the proposed waypoint generator compare to static and sub-sampled waypoint methods in terms of efficiency and robustness?
• RQ5 Can the integration of model-based global guidance with DRL local planning achieve higher success rates in complex, dynamic scenarios?

Key findings

• The proposed ST-WP approach achieved a 100% success rate in scenarios with 20 obstacles and velocities of 0.3 m/s and 0.5 m/s, significantly outperforming SH-WP (over 60%) and SUB-WP (over 35%).
• At 0.5 m/s obstacle speed, the success rate of SH-WP and SUB-WP dropped to over 50% and over 10%, respectively, while ST-WP maintained 100% success.
• The ST-WP method recorded only 11 collisions at 0.3 m/s and under 10 at 0.5 m/s, compared to over 25 collisions for both baseline methods.
• The ST-WP approach demonstrated slightly better efficiency than the baseline methods, with the SUB-WP method being the least efficient due to redundant traversal of all subgoals.
• The ST-WP method maintained robust and direct trajectories even at high obstacle speeds, while SH-WP and SUB-WP generated roundabout paths, especially near the 11,11 coordinate point.
• The use of dynamic obstacle modeling in the state-time space enabled the system to maintain path validity and safety despite high obstacle unpredictability.