[Paper Review] Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?

Ofir Nachum, Haoran Tang+4 more|arXiv (Cornell University)|Sep 23, 2019

Reinforcement Learning in RoboticsComputer Science40 references51 citations

TL;DR

The paper empirically analyzes HRL and finds that most benefits arise from improved exploration rather than easier policy learning or semantic action representations. It then proposes non-hierarchical exploration methods that match HRL performance.

ABSTRACT

Hierarchical reinforcement learning has demonstrated significant success at solving difficult reinforcement learning (RL) tasks. Previous works have motivated the use of hierarchy by appealing to a number of intuitive benefits, including learning over temporally extended transitions, exploring over temporally extended periods, and training and exploring in a more semantically meaningful action space, among others. However, in fully observed, Markovian settings, it is not immediately clear why hierarchical RL should provide benefits over standard "shallow" RL architectures. In this work, we isolate and evaluate the claimed benefits of hierarchical RL on a suite of tasks encompassing locomotion, navigation, and manipulation. Surprisingly, we find that most of the observed benefits of hierarchy can be attributed to improved exploration, as opposed to easier policy learning or imposed hierarchical structures. Given this insight, we present exploration techniques inspired by hierarchy that achieve performance competitive with hierarchical RL while at the same time being much simpler to use and implement.

Motivation & Objective

• Motivate the study of why hierarchical reinforcement learning (HRL) helps in complex tasks.
• Isolate and evaluate the purported benefits of hierarchy across locomotion, navigation, and manipulation tasks.
• Determine whether improvements come from training with temporally extended actions, exploration, or semantic representations.
• Assess whether non-hierarchical methods can match HRL performance by leveraging exploration strategies inspired by hierarchy.

Proposed method

• Empirically evaluate two HRL paradigms (options framework and goal-conditioned HIRO) on four locomotion/navigation/manipulation tasks (AntMaze, AntPush, AntBlock, AntBlockMaze).
• Isolate temporal abstraction effects by decoupling train horizon (c_train) and exploration horizon (c_expl).
• Compare HRL to non-hierarchical agents trained with multi-step rewards and to shadow agents trained on HRL-collected data.
• Propose and test two exploration strategies inspired by HRL (Explore & Exploit and Switching Ensemble) that do not use explicit hierarchy.
• Use ablation experiments to disentangle exploration versus training representations in HRL performance.

Experimental results

Research questions

• RQ1 Does temporally extended training or exploration explain the empirical gains of HRL in the considered tasks?
• RQ2 Are the benefits of high-level action representations (semantic training) essential for HRL performance?
• RQ3 Can non-hierarchical agents match HRL performance if given HRL-like exploration or multi-step rewards?
• RQ4 Do HRL-inspired exploration strategies improve non-hierarchical agents to HRL-level performance?

Key findings

• Most HRL benefits stem from improved exploration rather than easier training or semantic action representations.
• Controlling for exploration shows multi-step rewards can replicate much of the HRL training benefit, making high-level action representations less critical.
• Non-hierarchical agents equipped with temporally extended exploration or goal-directed exploration can match HRL performance on several tasks.
• Two non-hierarchical exploration methods (Explore & Exploit and Switching Ensemble) achieve HRL-like performance, underscoring exploration as the key factor.
• Explicit hierarchical structures are not strictly necessary for strong performance; exploration strategies inspired by HRL suffice in the tested environments.