[Paper Review] Stochastic Neural Networks for Hierarchical Reinforcement Learning

Carlos Florensa, Yan Duan+1 more|arXiv (Cornell University)|Apr 10, 2017

Reinforcement Learning in RoboticsComputer Science36 references196 citations

TL;DR

This paper presents a framework that pre-trains a span of diverse skills using Stochastic Neural Networks with an information-theoretic regularizer, then learns high-level policies on top of these skills to solve downstream tasks with sparse rewards.

ABSTRACT

Deep reinforcement learning has achieved many impressive results in recent years. However, tasks with sparse rewards or long horizons continue to pose significant challenges. To tackle these important problems, we propose a general framework that first learns useful skills in a pre-training environment, and then leverages the acquired skills for learning faster in downstream tasks. Our approach brings together some of the strengths of intrinsic motivation and hierarchical methods: the learning of useful skill is guided by a single proxy reward, the design of which requires very minimal domain knowledge about the downstream tasks. Then a high-level policy is trained on top of these skills, providing a significant improvement of the exploration and allowing to tackle sparse rewards in the downstream tasks. To efficiently pre-train a large span of skills, we use Stochastic Neural Networks combined with an information-theoretic regularizer. Our experiments show that this combination is effective in learning a wide span of interpretable skills in a sample-efficient way, and can significantly boost the learning performance uniformly across a wide range of downstream tasks.

Motivation & Objective

• Motivate and address learning in tasks with sparse rewards and long horizons.
• Propose a pre-training stage that discovers a wide span of useful skills with minimal task-specific guidance.
• Introduce Stochastic Neural Networks (SNNs) to represent multi-modal policies and enable skill diversity.
• Incorporate an information-theoretic regularizer to encourage distinct, interpretable skills.
• Train high-level policies on top of the learned skills to accelerate downstream learning.

Proposed method

• Use a pre-training environment with a proxy reward to guide skill discovery.
• Model policies with Stochastic Neural Networks that incorporate latent variables as input to a deterministic feed-forward net.
• Integrate latent variables via bilinear interaction to enhance multimodality and skill diversity.
• Add a mutual information-based bonus to maximize the informativeness of latent codes with respect to a relevant state component.
• Freeze the learned low-level skills and train a high-level manager to select a skill for a fixed horizon, enabling hierarchical control.
• Optimize policies with Trust Region Policy Optimization (TRPO) in both pre-training and downstream tasks.

Experimental results

Research questions

• RQ1 Can SNNs with an MI bonus reliably produce a large, interpretable span of skills?
• RQ2 Does pre-training with proxy rewards improve exploration and sample efficiency in downstream sparse-reward tasks?
• RQ3 Does a hierarchical policy using learned skills accelerate learning in mazes and gathering tasks compared with baselines?
• RQ4 To what extent does bilinear integration of latent variables enhance skill diversity and downstream performance?

Key findings

• SNNs with bilinear integration and MI regularization yield a broad set of interpretable skills.
• Hierarchical use of learned skills improves exploration in sparse-reward environments compared with non-hierarchical baselines.
• Pre-training with proxy rewards plus MI bonus accelerates learning in mazes and food gathering tasks across multiple runs.
• The approach reduces sample complexity by sharing low-level policies while learning high-level task-specific managers.
• Experiments show stronger performance and reduced variance in some downstream tasks when using MI-regularized SNNs.