[Paper Review] One-Shot Reinforcement Learning for Robot Navigation with Interactive Replay

Jake Bruce, Niko Sünderhauf+3 more|arXiv (Cornell University)|Nov 28, 2017

Reinforcement Learning in RoboticsComputer Science30 references53 citations

TL;DR

The paper presents offline learning for robot navigation using interactive replay from a single real-world traversal, a fixed pre-trained visual encoder, and stochastic observation augmentation to achieve zero-shot transfer without fine-tuning.

ABSTRACT

Recently, model-free reinforcement learning algorithms have been shown to solve challenging problems by learning from extensive interaction with the environment. A significant issue with transferring this success to the robotics domain is that interaction with the real world is costly, but training on limited experience is prone to overfitting. We present a method for learning to navigate, to a fixed goal and in a known environment, on a mobile robot. The robot leverages an interactive world model built from a single traversal of the environment, a pre-trained visual feature encoder, and stochastic environmental augmentation, to demonstrate successful zero-shot transfer under real-world environmental variations without fine-tuning.

Motivation & Objective

• Reduce real-world interaction by learning from a single environment traversal via an interactive world model.
• Leverage a fixed, pre-trained visual encoder to prevent overfitting to a small training set.
• Augment training with stochastic observations to improve generalization and transfer.
• Evaluate bootstrapped Q-learning for robust, transferable navigation performance.

Proposed method

• Construct a virtual training environment from a single traversal by building a pose graph and their interactive replay buffer.
• Use a fixed 2048-dim ResNet-50 visual encoder to produce an 8192-dim observation vector per timestep.
• Augment observations by sampling from distributions around the true pose to create stochastic observations.
• Train a bootstrapped double dueling Q-learning architecture with NQ heads and Nw parallel workers.
• Compare with A2C and single-head n-step Q-learning to assess learning and transfer performance.

Experimental results

Research questions

• RQ1 Can navigation to a fixed goal be learned with zero fine-tuning using offline interactive replay from one traversal?
• RQ2 How do pre-trained visual features and stochastic observations affect learning and transfer to unseen environmental variations?
• RQ3 Does bootstrapped Q-learning outperform other RL methods in this robotics navigation setting?
• RQ4 What is the impact of multiple Q-heads and parallel workers on training stability and performance?

Key findings

• Bootstrapped Q-learning achieved the best performance among the tested RL methods; other algorithms failed to reliably solve the task.
• Augmenting the environment with stochastic observations significantly improves training and validation performance.
• Using a pre-trained visual encoder (fixed during training) enhances transfer and reduces computational requirements.
• Stochastic observations improved transfer to the validation environment, indicating better generalization to unseen variations.
• Pre-trained features plus stochastic augmentation yielded dramatically better training and transfer outcomes compared to training from scratch.