[Paper Review] Curiosity-driven Exploration for Mapless Navigation with Deep Reinforcement Learning

Oleksii Zhelo, Jingwei Zhang+3 more|arXiv (Cornell University)|Apr 2, 2018

Robotics and Sensor-Based Localization4 references83 citations

TL;DR

The paper augments DRL mapless navigation with intrinsic curiosity via an Intrinsic Curiosity Module (ICM), showing improved learning efficiency and generalization to unseen maps.

ABSTRACT

This paper investigates exploration strategies of Deep Reinforcement Learning (DRL) methods to learn navigation policies for mobile robots. In particular, we augment the normal external reward for training DRL algorithms with intrinsic reward signals measured by curiosity. We test our approach in a mapless navigation setting, where the autonomous agent is required to navigate without the occupancy map of the environment, to targets whose relative locations can be easily acquired through low-cost solutions (e.g., visible light localization, Wi-Fi signal localization). We validate that the intrinsic motivation is crucial for improving DRL performance in tasks with challenging exploration requirements. Our experimental results show that our proposed method is able to more effectively learn navigation policies, and has better generalization capabilities in previously unseen environments. A video of our experimental results can be found at https://goo.gl/pWbpcF.

Motivation & Objective

• Motivate effective exploration in mapless robot navigation where environment maps are unavailable.
• Integrate intrinsic motivation with extrinsic rewards to guide policy learning.
• Evaluate generalization of learned policies to unseen environments and structures.

Proposed method

• Extend A3C reinforcement learning with a combined external and intrinsic reward signal R = Re + liRi.
• Implement Intrinsic Curiosity Module (ICM) comprising feature extractor φ, inverse model ψi and forward model ψf to produce intrinsic reward Ri from prediction error.
• Train with asynchronous actors (A3C) using a mix of external and intrinsic rewards to encourage exploration of novel states.
• Use a laser range sensor and relative goal pose as state; actions are discrete (go straight, turn left, turn right).
• ICM loss combines inverse model cross-entropy and forward model regression to shape φ and encourage useful representations.

Experimental results

Research questions

• RQ1 Does intrinsic curiosity improve sample efficiency and convergence of DRL for mapless navigation?
• RQ2 How does intrinsic motivation affect generalization to unseen environments with different layouts?
• RQ3 Under what conditions (with/without LSTM, with/without entropy) does curiosity-driven exploration outperform standard exploration?
• RQ4 What are the trade-offs between intrinsic and extrinsic rewards in guiding navigation policies?

Key findings

• ICM-based exploration yields higher success rates and often shorter paths than entropy-only or no-curiosity baselines on the training map.
• On Map1, ICM achieved 98.7% success with 91.2 steps on average (std 62.5), compared to 88.3% and 173.1 steps for the strongest non-ICM baseline.
• Combining ICM with entropy further improves performance and stability, achieving 100% success with 75.2 steps on average (std 52.1) in Map1 with LSTM.
• ICM improves generalization to unseen maps (Map2–4), particularly in more challenging layouts, where ICM+Entropy often yields better or comparable success and lower step counts than alternatives.
• Results suggest curiosity helps escape local minima and accelerates learning by driving the agent toward novel and informative states.