[Paper Review] Learning Abstract Options

Matthew Riemer, Miao Liu+1 more|arXiv (Cornell University)|Jan 1, 2018

Reinforcement Learning in Robotics30 citations

TL;DR

This paper proposes a deep hierarchical option-critic framework that enables end-to-end learning of temporally abstract options across multiple time resolutions without intrinsic rewards or subgoals. By extending policy gradient theorems to arbitrarily deep option hierarchies, the method learns internal policies, termination conditions, and compositional structures, demonstrating improved sample efficiency in both discrete and continuous control environments.

ABSTRACT

Building systems that autonomously create temporal abstractions from data is a key challenge in scaling learning and planning in reinforcement learning. One popular approach for addressing this challenge is the options framework (Sutton et al., 1999). However, only recently in (Bacon et al., 2017) was a policy gradient theorem derived for online learning of general purpose options in an end to end fashion. In this work, we extend previous work on this topic that only focuses on learning a two-level hierarchy including options and primitive actions to enable learning simultaneously at multiple resolutions in time. We achieve this by considering an arbitrarily deep hierarchy of options where high level temporally extended options are composed of lower level options with finer resolutions in time. We extend results from (Bacon et al., 2017) and derive policy gradient theorems for a deep hierarchy of options. Our proposed hierarchical option-critic architecture is capable of learning internal policies, termination conditions, and hierarchical compositions over options without the need for any intrinsic rewards or subgoals. Our empirical results in both discrete and continuous environments demonstrate the efficiency of our framework.

Motivation & Objective

• To address the challenge of scaling reinforcement learning by enabling autonomous learning of temporal abstractions across multiple time resolutions.
• To extend the options framework beyond two-level hierarchies to support arbitrarily deep compositions of options.
• To derive policy gradient theorems for deep option hierarchies, enabling end-to-end training of internal policies and termination conditions.
• To eliminate reliance on intrinsic rewards or manually specified subgoals in learning hierarchical options.
• To demonstrate the efficiency and scalability of the proposed framework in diverse environments.

Proposed method

• The paper extends the policy gradient theorem from Bacon et al. (2017) to support deep hierarchies of options, enabling gradient-based learning across multiple levels of temporal abstraction.
• It introduces a hierarchical option-critic architecture that jointly learns option policies, termination conditions, and compositional structures across levels.
• The framework supports arbitrary depth in the hierarchy, where high-level options are composed of lower-level options with finer temporal resolution.
• The method operates in an end-to-end fashion, training all components—policies, termination conditions, and compositions—without auxiliary supervision or intrinsic rewards.
• The architecture uses a credit assignment mechanism that propagates gradients through the hierarchical option structure to optimize all components simultaneously.

Experimental results

Research questions

• RQ1 Can a hierarchical option-critic framework learn temporally extended options across multiple time resolutions without relying on intrinsic rewards or subgoals?
• RQ2 How does the proposed policy gradient theorem generalize to deep option hierarchies beyond two-level structures?
• RQ3 What is the impact of hierarchical composition on sample efficiency and learning performance in complex environments?
• RQ4 How does the framework scale across different levels of temporal abstraction in both discrete and continuous control tasks?

Key findings

• The proposed framework successfully learns internal policies, termination conditions, and hierarchical compositions across multiple time resolutions without intrinsic rewards or subgoals.
• The method achieves improved sample efficiency in both discrete and continuous control environments due to the use of multi-resolution temporal abstractions.
• The deep hierarchical structure enables more effective credit assignment and faster convergence compared to flat or shallow option structures.
• Empirical results confirm that the framework generalizes across diverse environments, demonstrating robustness and scalability in hierarchical reinforcement learning.