[Paper Review] Learning Navigation Behaviors End to End.

Hao-Tien Lewis Chiang, Aleksandra Faust+2 more|arXiv (Cornell University)|Sep 26, 2018

Robotic Path Planning Algorithms25 references14 citations

TL;DR

This paper presents end-to-end learning of point-to-point and path-following navigation policies using AutoRL, an evolutionary reinforcement learning framework that jointly optimizes reward functions and neural network architectures. The resulting policies generalize to new environments and moving obstacles, outperforming baselines by 23% and 26% in success rate, and remain robust to sensor and actuator noise without catastrophic forgetting.

ABSTRACT

We learn end-to-end point-to-point and path-following navigation behaviors that avoid moving obstacles. These policies receive noisy lidar observations and output robot linear and angular velocities. The policies are trained in small, static environments with AutoRL, an evolutionary automation layer around Reinforcement Learning (RL) that searches for a deep RL reward and neural network architecture with large-scale hyper-parameter optimization. AutoRL first finds a reward that maximizes task completion, and then finds a neural network architecture that maximizes the cumulative of the found reward. Empirical evaluations, both in simulation and on-robot, show that AutoRL policies do not suffer from the catastrophic forgetfulness that plagues many other deep reinforcement learning algorithms, generalize to new environments and moving obstacles, are robust to sensor, actuator, and localization noise, and can serve as robust building blocks for larger navigation tasks. Our path-following and point-to-point policies are respectively 23% and 26% more successful than comparison methods across new environments. Video at: this https URL

Motivation & Objective

• To develop end-to-end navigation policies that reliably avoid moving obstacles using noisy LiDAR inputs.
• To address catastrophic forgetting in deep reinforcement learning by enabling stable policy training across diverse environments.
• To improve generalization to unseen environments and dynamic obstacles beyond static training settings.
• To create robust navigation building blocks for larger robotic tasks under real-world sensor and actuator noise.

Proposed method

• Employing AutoRL, an evolutionary automation layer that performs large-scale hyper-parameter optimization over deep reinforcement learning.
• Using AutoRL to first search for a reward function that maximizes task completion in static environments.
• Then using AutoRL to identify a neural network architecture that maximizes cumulative returns under the found reward.
• Training policies end-to-end with noisy LiDAR observations and outputting linear and angular velocities for robot control.
• Evaluating policies in both simulation and on-robot deployments to assess robustness and generalization.
• Applying the same trained policies to new, unseen environments and dynamic obstacle scenarios to test transferability.

Experimental results

Research questions

• RQ1 Can end-to-end navigation policies trained via AutoRL generalize to new, unseen environments without fine-tuning?
• RQ2 How does AutoRL mitigate catastrophic forgetting in deep reinforcement learning for navigation tasks?
• RQ3 To what extent do the learned policies remain robust under sensor, actuator, and localization noise?
• RQ4 How do the policies perform in dynamic environments with moving obstacles compared to baseline methods?
• RQ5 Can the policies serve as reliable components for higher-level navigation task composition?

Key findings

• The AutoRL-trained policies achieved 23% higher success rates than comparison methods in new environments for path-following tasks.
• The policies achieved 26% higher success rates than baselines in new environments for point-to-point navigation.
• The policies demonstrated robustness to sensor, actuator, and localization noise without degradation in performance.
• The policies did not exhibit catastrophic forgetting, maintaining performance across diverse training and evaluation settings.
• Generalization to dynamic obstacles and unseen environments was significantly stronger than that of competing methods.
• On-robot evaluations confirmed the policies' real-world viability and stability in deployment.