[Paper Review] Virtual-to-real Deep Reinforcement Learning: Continuous Control of Mobile Robots for Mapless Navigation

Lei Tai, Giuseppe Paolo+1 more|arXiv (Cornell University)|Mar 1, 2017

Robotic Path Planning Algorithms23 references99 citations

TL;DR

The paper presents a mapless motion planner trained end-to-end with asynchronous deep reinforcement learning using only 10-dimensional sparse laser range findings and target relative position, and demonstrates transfer from virtual to real robots without fine-tuning.

ABSTRACT

We present a learning-based mapless motion planner by taking the sparse 10-dimensional range findings and the target position with respect to the mobile robot coordinate frame as input and the continuous steering commands as output. Traditional motion planners for mobile ground robots with a laser range sensor mostly depend on the obstacle map of the navigation environment where both the highly precise laser sensor and the obstacle map building work of the environment are indispensable. We show that, through an asynchronous deep reinforcement learning method, a mapless motion planner can be trained end-to-end without any manually designed features and prior demonstrations. The trained planner can be directly applied in unseen virtual and real environments. The experiments show that the proposed mapless motion planner can navigate the nonholonomic mobile robot to the desired targets without colliding with any obstacles.

Motivation & Objective

• Develop a mapless motion planner for nonholonomic mobile robots that uses sparse range data and relative target position as input.
• Train the planner end-to-end with asynchronous deep reinforcement learning to output continuous steering commands.
• Demonstrate transferability of the learned policy from virtual simulations to real-world robots without fine-tuning.
• Evaluate robustness and compare with a traditional map-based planner under virtual and real conditions.

Proposed method

• Formulate as a continuous-control DRL problem with state x_t consisting of 10-dim sparse laser findings, previous velocity, and relative target position.
• Use asynchronous DDPG (ADDPG) to train actor and critic networks in parallel sample collection threads.
• Network outputs are continuous linear and angular velocities constrained to (-1,1) for angular and (0,1) for linear via tanh and sigmoid activations.
• Train in virtual environments (V-REP) with 2 indoor layouts, randomize targets, and optimize a reward combining arrival, collision, and progress to the target.
• Reward: r = r_arrive if close to target, r_collision if collision, else c_r(d_{t-1}-d_t) where d_t is distance to target.
• Observation input is merged into a 14-dim vector (10-d laser findings, 2-d previous velocity, 2-d relative target).
• Employ a 3-layer fully connected critic and actor networks with 512-node layers for the policy.
• Use an Adam optimizer and cap outputs to realistic robot limits (e.g., max linear 0.5 m/s, max angular 1 rad/s).

Experimental results

Research questions

• RQ1 Can a mapless planner learn to navigate a nonholonomic differential-drive robot using only sparse range data and target position?
• RQ2 Does an asynchronous DRL approach enable efficient training for continuous control in this setting?
• RQ3 Can the learned policy transfer from virtual simulation to real-world robots without fine-tuning?
• RQ4 How does the mapless planner compare to a traditional map-based planner in terms of robustness and speed in unseen environments?

Key findings

• The mapless planner trained in simulation using 10-dim sparse laser readings achieves collision-free navigation to targets in unseen virtual environments.
• The ADDPG method accelerates training by increasing sample throughput and improving Q-value convergence compared to standard DDPG.
• In real-world tests, the Env-2 trained model completed navigation tasks where the 10-dim Move Base failed or required human intervention, demonstrating transfer without fine-tuning.
• The mapless planner runs at about 1 ms per decision, significantly faster than the map-based baseline.
• The learned policies show robustness in complex indoor settings and can be extended to low-cost sensors with sparse observations.
• A recovery behavior emerges naturally (rotating recovery) when needed, with no observed collisions during experiments.