[Paper Review] Learning to Navigate in Complex Environments

Piotr Mirowski, Razvan Pascanu+10 more|arXiv (Cornell University)|Nov 11, 2016

Reinforcement Learning in Robotics366 citations

TL;DR

The paper trains end-to-end reinforcement learning agents for navigation in 3D mazes using auxiliary tasks (depth prediction and loop closure) to boost data efficiency and performance, achieving near-human results in dynamic goal settings.

ABSTRACT

Learning to navigate in complex environments with dynamic elements is an important milestone in developing AI agents. In this work we formulate the navigation question as a reinforcement learning problem and show that data efficiency and task performance can be dramatically improved by relying on additional auxiliary tasks leveraging multimodal sensory inputs. In particular we consider jointly learning the goal-driven reinforcement learning problem with auxiliary depth prediction and loop closure classification tasks. This approach can learn to navigate from raw sensory input in complicated 3D mazes, approaching human-level performance even under conditions where the goal location changes frequently. We provide detailed analysis of the agent behaviour, its ability to localise, and its network activity dynamics, showing that the agent implicitly learns key navigation abilities.

Motivation & Objective

• Motivate learning to navigate as an RL problem without explicit SLAM/MSM mapping.
• Improve data efficiency and performance by introducing auxiliary tasks that leverage multimodal inputs.
• Demonstrate that auxiliary depth prediction and loop closure classification help agents navigate in dynamic mazes.
• Analyze how auxiliary tasks influence internal representations and localization abilities.
• Provide insights into how memory and representation learning emerge during navigation tasks.

Proposed method

• Use actor-critic (A3C) with a convolutional encoder followed by LSTM-based memory.
• Incorporate auxiliary depth prediction to reconstruct a low-resolution depth map from RGB inputs.
• Incorporate loop-closure prediction to detect revisits using integrated 2D velocity information.
• Two depth formulations: predict depth from convolutional features (D1) or from top LSTM layer (D2); compare against a loop-closure loss (L).
• Train with a weighted combination of RL loss, depth loss (βd1, βd2), and loop-closure loss (βl).
• Evaluate across five 3D maze environments with static and random goal placements, using Nav A3C architectures with varying memory and inputs.

Experimental results

Research questions

• RQ1 Can auxiliary tasks improve data efficiency and performance in end-to-end navigation policies?
• RQ2 Does depth prediction, as a self-supervised auxiliary task, help learn geometry and obstacle avoidance for navigation?
• RQ3 Does loop-closure prediction encourage better spatial localization and memory integration for navigation in dynamic mazes?
• RQ4 Which auxiliary task configuration (D1, D2, L, or combinations) yields the best navigation performance and localization?
• RQ5 How does memory architecture (stacked LSTM with velocity, action, and reward inputs) affect navigation in complex mazes?

Key findings

• Auxiliary tasks substantially speed up learning and improve performance across mazes, especially static ones.
• Depth prediction from the policy’s LSTM (D2) yields strong navigation performance and localization benefits.
• Classification formulation of depth prediction converges faster than regression in this setting.
• Loop-closure prediction complements depth, aiding velocity integration and spatial reasoning; combined losses often outperform single tasks.
• Nav A3C with auxiliary losses approaches human-level performance on static mazes and achieves substantial scores on dynamic/random goal mazes.
• Position decoders trained on internal representations indicate improved localization correlating with higher task rewards.