[Paper Review] Hierarchical Reinforcement Learning By Discovering Intrinsic Options

Jesse Zhang, Haonan Yu+1 more|arXiv (Cornell University)|Jan 16, 2021

Reinforcement Learning in Robotics41 references24 citations

TL;DR

HIDIO learns task-agnostic intrinsic options in a self-supervised manner and jointly uses them to solve sparse-reward tasks, achieving higher sample efficiency than baselines.

ABSTRACT

We propose a hierarchical reinforcement learning method, HIDIO, that can learn task-agnostic options in a self-supervised manner while jointly learning to utilize them to solve sparse-reward tasks. Unlike current hierarchical RL approaches that tend to formulate goal-reaching low-level tasks or pre-define ad hoc lower-level policies, HIDIO encourages lower-level option learning that is independent of the task at hand, requiring few assumptions or little knowledge about the task structure. These options are learned through an intrinsic entropy minimization objective conditioned on the option sub-trajectories. The learned options are diverse and task-agnostic. In experiments on sparse-reward robotic manipulation and navigation tasks, HIDIO achieves higher success rates with greater sample efficiency than regular RL baselines and two state-of-the-art hierarchical RL methods.

Motivation & Objective

• Motivate automatic task decomposition for sparse-reward tasks without manual option design.
• Learn a two-level hierarchy where a scheduler selects latent options and a worker executes actions.
• Encourage diverse and task-agnostic low-level options through intrinsic entropy objectives.
• Demonstrate that self-supervised option discovery improves success rates and efficiency across robotic tasks.

Proposed method

• Two-level HIDIO architecture with a scheduler (high level) that outputs latent options u every K steps and a worker (low level) that acts conditioned on u.
• Option discovery via an intrinsic reward that maximizes a log-q posterior p(u|trajectory) using a discriminator qψ.
• Lower bound optimization replacing the intractable posterior with a parameterized discriminator.
• A meta-MDP is defined so the worker observes option sub-trajectories, enabling effective discrimination and credit assignment.
• Worker objective combines intrinsic reward with a max-entropy RL objective (SAC) for stable learning.
• Discriminator instantiations for sub-trajectory features (State, Action, StateDiff, StateAction, StateConcat, ActionConcat) are evaluated.
• Off-policy training with replay buffers, intrinsic reward relabeling, and optional importance corrections to improve sample efficiency.

Experimental results

Research questions

• RQ1 Can task-agnostic intrinsic options learned in a self-supervised way improve performance on sparse-reward tasks?
• RQ2 How should the discriminator operate on option sub-trajectories to effectively discover diverse options?
• RQ3 Does joint training of the scheduler and worker outperform pre-training the worker with fixed options?
• RQ4 What is the impact of different short-sightedness and option lengths on learning and performance?
• RQ5 How do HIDIO’s intrinsic options compare to existing hierarchical RL methods on manipulation and navigation tasks?

Key findings

• HIDIO achieves higher final success rates and better sample efficiency than SAC, SAC+ActRepeat, HIRO, and HiPPO across four diverse tasks.
• Discriminator feature choices like Action, StateAction, and StateDiff generally yield stronger performance.
• Jointly training the scheduler and worker outperforms pre-training of the worker with fixed options on challenging tasks.
• Learned options behave as low-level navigation and manipulation skills that facilitate exploration and task completion.
• HIDIO can solve difficult sparse-reward tasks where baselines struggle, demonstrating the value of intrinsic, short-term option discovery.