[Paper Review] Learning with Stochastic Guidance for Navigation

Linhai Xie, Yishu Miao+6 more|arXiv (Cornell University)|Nov 27, 2018

Reinforcement Learning in Robotics8 references4 citations

TL;DR

This paper proposes a stochastic guidance framework that enables deep reinforcement learning agents to dynamically switch between high-variance exploration (via DDPG) and low-variance heuristic controllers (e.g., PID, obstacle avoidance) during training. By jointly training a stochastic switch with DDPG using REINFORCE, the method significantly reduces training variance, accelerates policy learning, and achieves superior navigation performance compared to state-of-the-art baselines, with successful zero-shot transfer to real-world environments.

ABSTRACT

Due to the sparse rewards and high degree of environment variation, reinforcement learning approaches such as Deep Deterministic Policy Gradient (DDPG) are plagued by issues of high variance when applied in complex real world environments. We present a new framework for overcoming these issues by incorporating a stochastic switch, allowing an agent to choose between high and low variance policies. The stochastic switch can be jointly trained with the original DDPG in the same framework. In this paper, we demonstrate the power of the framework in a navigation task, where the robot can dynamically choose to learn through exploration, or to use the output of a heuristic controller as guidance. Instead of starting from completely random moves, the navigation capability of a robot can be quickly bootstrapped by several simple independent controllers. The experimental results show that with the aid of stochastic guidance we are able to effectively and efficiently train DDPG navigation policies and achieve significantly better performance than state-of-the-art baselines models.

Motivation & Objective

• To address the high variance and poor sample efficiency of DDPG in complex, high-dimensional navigation environments with sparse rewards.
• To enable faster and more stable training of DDPG policies by incorporating low-variance heuristic controllers as dynamic guidance.
• To allow the DDPG policy to eventually operate independently after being bootstrapped by heuristic guidance.
• To enable zero-shot transfer of policies trained in simulation to real-world robotic navigation tasks.

Proposed method

• A stochastic switch mechanism is introduced that probabilistically selects actions from either the DDPG policy or independent heuristic controllers (PID and obstacle avoidance).
• The switch is trained via the REINFORCE policy gradient algorithm to maximize cumulative return, enabling context-dependent selection of controllers.
• The DDPG agent learns from actions selected by the stochastic switch rather than from its own policy output, reducing gradient variance during early training.
• The framework uses a 3-layer convolutional neural network to process depth camera observations and generate geometric representations for state input.
• The switch function is implemented as a neural network conditioned on full sensor inputs, allowing adaptive controller selection based on environmental context.
• The system supports progressive deactivation of heuristic controllers once the DDPG policy achieves sufficient performance, enabling autonomous operation.

Experimental results

Research questions

• RQ1 Can a stochastic switch mechanism improve the sample efficiency and training stability of DDPG in complex navigation tasks?
• RQ2 How does dynamic switching between exploration and heuristic guidance affect final navigation performance compared to baseline DDPG?
• RQ3 To what extent can a DDPG policy trained with stochastic guidance generalize to real-world environments without further fine-tuning?
• RQ4 Can the DDPG policy be effectively tested in isolation after training, indicating successful internalization of guidance?
• RQ5 How does the choice of heuristic controllers influence the final performance and generalization of the DDPG policy?

Key findings

• The proposed framework achieves significantly better navigation performance than state-of-the-art DDPG baselines, with faster convergence and lower training variance.
• The model successfully transfers from simulation to a real-world environment with a Turtlebot and depth camera, achieving collision-free navigation across multiple goals without retraining.
• The DDPG policy can be tested in isolation after training, demonstrating that it has learned to perform navigation autonomously without relying on external controllers.
• The framework shows strong generalization, benefiting from multiple independent controllers (e.g., PID and OA), with performance improving as more heuristics are incorporated.
• The system can automatically reduce reliance on heuristic controllers when the DDPG policy becomes competent, indicating effective internalization of guidance.
• The ablation study confirms that the stochastic switch outperforms both uniform random switching and argmax switching, validating the effectiveness of the learned switching policy.