[Paper Review] Hierarchical Reinforcement Learning with Hindsight

Andrew Levy, Robert W. Platt+1 more|arXiv (Cornell University)|May 21, 2018

Reinforcement Learning in Robotics16 references42 citations

TL;DR

The paper proposes a method that learns temporally extended actions at multiple abstraction levels by combining universal value functions with hindsight learning, enabling parallel learning across time scales and improving sample efficiency in both discrete and continuous tasks.

ABSTRACT

Reinforcement Learning (RL) algorithms can suffer from poor sample efficiency when rewards are delayed and sparse. We introduce a solution that enables agents to learn temporally extended actions at multiple levels of abstraction in a sample efficient and automated fashion. Our approach combines universal value functions and hindsight learning, allowing agents to learn policies belonging to different time scales in parallel. We show that our method significantly accelerates learning in a variety of discrete and continuous tasks.

Motivation & Objective

• Address sample inefficiency in RL with delayed and sparse rewards.
• Enable learning of temporally extended actions at multiple abstraction levels.
• Develop a method that learns policies across different time scales in parallel.
• Integrate universal value functions with hindsight learning to facilitate multitimescale learning.

Proposed method

• Use universal value functions to represent value across different goals and time scales.
• Incorporate hindsight learning to reframe past experiences with alternative goals for richer learning signals.
• Enable parallel learning of policies at multiple temporal horizons within a single framework.
• Leverage a hierarchical structure to learn actions at different levels of abstraction in a sample-efficient manner.
• Apply the approach to both discrete and continuous control tasks to demonstrate generality.

Experimental results

Research questions

• RQ1 Can universal value functions combined with hindsight learning support learning of temporally extended actions across multiple time scales?
• RQ2 Does the proposed hierarchical approach improve sample efficiency compared to flat RL baselines in both discrete and continuous tasks?
• RQ3 Can policies at different temporal horizons be learned in parallel without interference?
• RQ4 How does hindsight-based relabeling influence learning speed and policy quality across hierarchical levels?
• RQ5 What are the practical benefits and limitations of automating multitimescale learning in RL?

Key findings

• The method accelerates learning in a variety of discrete tasks.
• The method accelerates learning in a variety of continuous tasks.
• Learning occurs across multiple time scales in parallel, improving sample efficiency.