[Paper Review] Learning Multi-Level Hierarchies with Hindsight

Andrew Levy, George Konidaris+2 more|arXiv (Cornell University)|Dec 4, 2017

Reinforcement Learning in Robotics75 citations

TL;DR

This paper introduces Hierarchical Actor-Critic (HAC), a hierarchical reinforcement learning framework that trains multiple levels of policies in parallel using hindsight action/goal transitions to overcome non-stationarity and sparse rewards, enabling efficient learning in continuous state/action spaces.

ABSTRACT

Hierarchical agents have the potential to solve sequential decision making tasks with greater sample efficiency than their non-hierarchical counterparts because hierarchical agents can break down tasks into sets of subtasks that only require short sequences of decisions. In order to realize this potential of faster learning, hierarchical agents need to be able to learn their multiple levels of policies in parallel so these simpler subproblems can be solved simultaneously. Yet, learning multiple levels of policies in parallel is hard because it is inherently unstable: changes in a policy at one level of the hierarchy may cause changes in the transition and reward functions at higher levels in the hierarchy, making it difficult to jointly learn multiple levels of policies. In this paper, we introduce a new Hierarchical Reinforcement Learning (HRL) framework, Hierarchical Actor-Critic (HAC), that can overcome the instability issues that arise when agents try to jointly learn multiple levels of policies. The main idea behind HAC is to train each level of the hierarchy independently of the lower levels by training each level as if the lower level policies are already optimal. We demonstrate experimentally in both grid world and simulated robotics domains that our approach can significantly accelerate learning relative to other non-hierarchical and hierarchical methods. Indeed, our framework is the first to successfully learn 3-level hierarchies in parallel in tasks with continuous state and action spaces.

Motivation & Objective

• Motivate the use of hierarchy to accelerate learning in sequential decision tasks.
• Develop a framework to learn multiple levels of policies in parallel despite non-stationary transitions.
• Propose mechanisms (hindsight action/goal transitions and subgoal testing) to enable stable parallel learning with sparse rewards.
• Demonstrate scalability to 2- and 3-level hierarchies in grid world and continuous robotics domains.

Proposed method

• Propose Hierarchical Actor-Critic (HAC), transforming a single UMDP into multiple nested UMDPs for each hierarchy level.
• Use goal-conditioned policies where each level outputs subgoals for the level below and ultimately primitive actions at the bottom level.
• Employ nested transition functions where higher levels’ transitions depend on the full lower-level policy hierarchy.
• Introduce hindsight action transitions to simulate the optimal lower-level hierarchy, stabilizing learning across levels.
• Introduce hindsight goal transitions to extend Hindsight Experience Replay to hierarchical settings for sparse rewards.
• Add subgoal testing transitions to ensure subgoals are achievable by the current lower-level policies and balance learning signals.

Experimental results

Research questions

• RQ1 Can HAC learn multiple levels of policies in parallel in both discrete and continuous domains?
• RQ2 Does HAC enable training of 3-level hierarchies in parallel, and how does it compare to 2-level and flat baselines?
• RQ3 Do hindsight action/goal transitions and subgoal testing transitions mitigate non-stationarity and improve learning efficiency?
• RQ4 How does HAC perform relative to HIRO on continuous robotics tasks?

Key findings

• HAC substantially outperformed flat agents across discrete and continuous tasks.
• A 3-level hierarchy learned in parallel outperformed a 2-level hierarchy, which in turn outperformed flat learning.
• HAC outperformed HIRO on three simulated robotics tasks in the experiments.
• Hindsight action and goal transitions, plus subgoal testing, enable stable parallel learning and mitigate issues from non-stationary transitions.