[Paper Review] End-to-End Navigation in Unknown Environments using Neural Networks

Arbaaz Khan, Clark Zhang+4 more|ArXiv.org|Jul 24, 2017

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes an end-to-end navigation policy using value iteration networks (VIN) enhanced with differentiable memory (LSTM) to solve long cul-de-sac navigation in unknown environments. By learning to represent past sensory history, the model generalizes to unseen cul-de-sac lengths up to 200 units after training on 20-unit paths, outperforming supervised and reinforcement learning baselines.

ABSTRACT

We investigate how a neural network can learn perception actions loops for navigation in unknown environments. Specifically, we consider how to learn to navigate in environments populated with cul-de-sacs that represent convex local minima that the robot could fall into instead of finding a set of feasible actions that take it to the goal. Traditional methods rely on maintaining a global map to solve the problem of over coming a long cul-de-sac. However, due to errors induced from local and global drift, it is highly challenging to maintain such a map for long periods of time. One way to mitigate this problem is by using learning techniques that do not rely on hand engineered map representations and instead output appropriate control policies directly from their sensory input. We first demonstrate that such a problem cannot be solved directly by deep reinforcement learning due to the sparse reward structure of the environment. Further, we demonstrate that deep supervised learning also cannot be used directly to solve this problem. We then investigate network models that offer a combination of reinforcement learning and supervised learning and highlight the significance of adding fully differentiable memory units to such networks. We evaluate our networks on their ability to generalize to new environments and show that adding memory to such networks offers huge jumps in performance

Motivation & Objective

• Address the challenge of navigating in unknown environments with convex local minima (e.g., cul-de-sacs) where traditional SLAM-based mapping fails due to drift.
• Investigate whether end-to-end deep reinforcement learning or supervised learning can solve sequential navigation without explicit map representations.
• Explore the role of memory in enabling generalization to longer, unseen cul-de-sacs beyond training distribution.
• Demonstrate that combining value iteration networks with differentiable memory units enables robust, generalizable navigation policies.

Proposed method

• Use a 2D grid world simulator with discrete actions (up, down, left, right) and depth sensor inputs to model robot perception.
• Train a value iteration network (VIN) to approximate value iteration for path planning using a learned reward prior.
• Augment VIN with a Long Short-Term Memory (LSTM) layer after the attention module to encode sequential sensory history.
• Train the VIN+LSTM model end-to-end using supervised supervision from an A* planner on cul-de-sac trajectories of fixed length (20 units).
• Evaluate generalization by testing on cul-de-sacs of increasing length (up to 500 units) without retraining.
• Compare performance against DQN, CNN, CNN+LSTM, and VIN with handcrafted partial maps to isolate the impact of memory and architecture.

Experimental results

Research questions

• RQ1 Can deep reinforcement learning solve end-to-end navigation in unknown environments with sparse rewards and complex local minima?
• RQ2 Can supervised learning with a fully convolutional network generalize to longer cul-de-sacs when sensor inputs are identical for entering and exiting?
• RQ3 Does adding an LSTM memory module to a value iteration network enable generalization to longer, unseen cul-de-sacs?
• RQ4 How does the performance of VIN+LSTM compare to a model with access to a hand-engineered partial map?
• RQ5 Can a differentiable memory mechanism like LSTM or DNC improve navigation robustness in sequential decision-making tasks?

Key findings

• DQN and standard CNN-based models failed entirely, achieving 0% success rate on cul-de-sac navigation due to sparse rewards and ambiguous sensor inputs.
• The CNN+LSTM model achieved 40% success but only generalized to cul-de-sacs up to 20 units long, indicating it learned a fixed turning distance rather than structural understanding.
• VIN with a handcrafted partial map achieved 100% success and generalized to cul-de-sacs up to 500 units, proving that memory of past observations is critical.
• VIN with an LSTM layer achieved 100% success and generalized to cul-de-sacs up to 190 units, significantly outperforming CNN+LSTM and demonstrating effective memory integration.
• The LSTM layer performed best when placed after the attention module in the VIN architecture, highlighting architectural sensitivity.
• Initial experiments with the Differentiable Neural Computer (DNC) showed improved performance over LSTM, suggesting potential for more advanced memory mechanisms in future work.