[Paper Review] Learning Navigation Behaviors End-to-End with AutoRL

Hao-Tien Lewis Chiang, Aleksandra Faust+2 more|arXiv (Cornell University)|Sep 26, 2018

Robotic Path Planning Algorithms39 references4 citations

TL;DR

This paper presents AutoRL, an automated reinforcement learning framework that jointly optimizes reward shaping and neural network architecture for end-to-end navigation policies. Trained in simulation on small static environments, the resulting point-to-point and path-following policies achieve 23% and 26% higher success rates than baselines in large, unseen real-world environments, demonstrating robustness to noise and dynamic obstacles.

ABSTRACT

We learn end-to-end point-to-point and path-following navigation behaviors that avoid moving obstacles. These policies receive noisy lidar observations and output robot linear and angular velocities. The policies are trained in small, static environments with AutoRL, an evolutionary automation layer around Reinforcement Learning (RL) that searches for a deep RL reward and neural network architecture with large-scale hyper-parameter optimization. AutoRL first finds a reward that maximizes task completion, and then finds a neural network architecture that maximizes the cumulative of the found reward. Empirical evaluations, both in simulation and on-robot, show that AutoRL policies do not suffer from the catastrophic forgetfulness that plagues many other deep reinforcement learning algorithms, generalize to new environments and moving obstacles, are robust to sensor, actuator, and localization noise, and can serve as robust building blocks for larger navigation tasks. Our path-following and point-to-point policies are respectively 23% and 26% more successful than comparison methods across new environments. Video at: https://youtu.be/0UwkjpUEcbI

Motivation & Objective

• To develop a scalable, automated method for training end-to-end navigation policies that generalize across diverse environments and sensor noise levels.
• To address the challenge of sparse rewards and poor generalization in deep reinforcement learning for robot navigation by automating reward and architecture search.
• To create navigation policies that are robust to sensor, actuator, and localization noise, and transfer effectively from simulation to real-world deployment.
• To enable policies that avoid both static and moving obstacles without requiring explicit obstacle velocity information or complex motion planning stacks.
• To reduce manual hyperparameter tuning by replacing expert-designed rewards and network architectures with automated, large-scale hyperparameter optimization.

Proposed method

• AutoRL uses gradient-free, evolutionary hyperparameter optimization (e.g., CMA-ES) to search over reward functions and neural network architectures in parallel.
• The method first optimizes a shaped reward function that maximizes task completion in simulation, then fixes the reward and searches for the optimal neural network architecture.
• Policies are trained using deep reinforcement learning (DDPG) with lidar observations and velocity outputs, operating at 5 Hz.
• The reward function is parameterized with learnable coefficients and includes terms for distance to goal, clearance from obstacles, and path deviation.
• Neural network architectures are defined by layer widths (e.g., [50, 20, 10] for P2P), and AutoRL searches over these hyperparameters to maximize cumulative return.
• Noise is injected during training to simulate real-world conditions: sensor noise (σ_Lidar = 0.3 m), actuator noise (σ_Speed = 0.1 m/s, σ_Turning = 0.1 rad/s), and localization noise (σ_Localize = 0.1 m).

Experimental results

Research questions

• RQ1 Can an automated method jointly optimize reward shaping and neural network architecture to improve generalization in end-to-end navigation policies?
• RQ2 To what extent do AutoRL-trained policies generalize to large, previously unseen environments with dynamic obstacles and sensor noise?
• RQ3 How does AutoRL compare to hand-tuned baselines (e.g., DWA, APF, BC) in success rate and robustness across simulation and real-robot deployment?
• RQ4 Can AutoRL policies avoid catastrophic forgetting and maintain performance under varying noise levels and environmental complexity?
• RQ5 Does the use of path guidance with PRM-generated waypoints improve performance in path-following tasks with moving obstacles?

Key findings

• AutoRL policies achieved 26% higher success rate in point-to-point navigation and 23% higher success rate in path-following tasks across three large, unseen real-world environments compared to baselines.
• The policies generalized robustly to environments 6.8 to 30.1 times larger than the training environment, with no fine-tuning.
• On a real Fetch robot, the policies successfully navigated over 80 meters of path with collision-free obstacle avoidance in dynamic environments.
• The policies demonstrated robustness to sensor, actuator, and localization noise, with no degradation in performance under high noise levels.
• AutoRL policies outperformed hand-tuned DDPG, APF, DWA, and behavior cloning baselines in both simulation and real-world evaluation.
• The method successfully avoided catastrophic forgetting, maintaining high performance across diverse training and evaluation conditions.