[Paper Review] Data-Efficient Hierarchical Reinforcement Learning

Ofir Nachum, Shixiang Gu+2 more|arXiv (Cornell University)|May 21, 2018

Reinforcement Learning in RoboticsComputer Science3 references265 citations

TL;DR

Proposes HIRO, a two-layer HRL agent trained off-policy with an off-policy correction, achieving high sample efficiency and strong performance on locomotion and object-interaction tasks.

ABSTRACT

Hierarchical reinforcement learning (HRL) is a promising approach to extend traditional reinforcement learning (RL) methods to solve more complex tasks. Yet, the majority of current HRL methods require careful task-specific design and on-policy training, making them difficult to apply in real-world scenarios. In this paper, we study how we can develop HRL algorithms that are general, in that they do not make onerous additional assumptions beyond standard RL algorithms, and efficient, in the sense that they can be used with modest numbers of interaction samples, making them suitable for real-world problems such as robotic control. For generality, we develop a scheme where lower-level controllers are supervised with goals that are learned and proposed automatically by the higher-level controllers. To address efficiency, we propose to use off-policy experience for both higher and lower-level training. This poses a considerable challenge, since changes to the lower-level behaviors change the action space for the higher-level policy, and we introduce an off-policy correction to remedy this challenge. This allows us to take advantage of recent advances in off-policy model-free RL to learn both higher- and lower-level policies using substantially fewer environment interactions than on-policy algorithms. We term the resulting HRL agent HIRO and find that it is generally applicable and highly sample-efficient. Our experiments show that HIRO can be used to learn highly complex behaviors for simulated robots, such as pushing objects and utilizing them to reach target locations, learning from only a few million samples, equivalent to a few days of real-time interaction. In comparisons with a number of prior HRL methods, we find that our approach substantially outperforms previous state-of-the-art techniques.

Motivation & Objective

• Motivate and develop general, data-efficient HRL that works with standard RL components.
• Learn lower-level policies guided by goals proposed automatically by higher-level controllers.
• Enable off-policy training for both hierarchy levels to improve sample efficiency.
• Introduce an off-policy correction to address non-stationarity of lower-level changes.
• Demonstrate strong performance on challenging simulated robotic tasks with limited interaction data.

Proposed method

• Two-layer hierarchy with a high-level policy (goals) and a low-level policy (actions).
• Lower level receives a goal g_t and incurs intrinsic reward r = -||s_t + g_t - s_{t+1}||_2; higher level optimizes over temporally-extended goals every c steps.
• Higher level experiences are relabeled (off-policy correction) to maximize the probability that past low-level actions would occur under the current lower-level controller, enabling off-policy learning.
• Both policies are trained with off-policy TD methods (TD3) using replay buffers.
• Goals are defined directly in raw state observations, avoiding learned embeddings or manual goal spaces.
• An eight-candidate relabeling procedure plus original and difference-based goals are used for high-level relabeling to approximate argmax of likelihood.

Experimental results

Research questions

• RQ1 Can a two-level HRL system trained off-policy with an off-policy correction learn complex tasks efficiently?
• RQ2 Does using raw state observations as goals for the lower-level policy improve learning speed and performance?
• RQ3 How does the proposed off-policy correction impact stability and sample efficiency versus naive off-policy HRL?
• RQ4 What is the comparative performance of HIRO on challenging locomotion and object-interaction tasks against prior HRL methods?

Key findings

• HIRO achieves strong performance across Ant Gather, Ant Maze, Ant Push, and Ant Fall tasks.
• On 10M steps, HIRO outperforms baselines including FuN variants, SNN4HRL, and VIME for all tasks; Ant Gather is the closest competitor with pre-training the lower level.
• HIRO demonstrates rapid learning, solving complex tasks after a few million environment steps (a few days of real-world interaction).
• Off-policy correction is crucial for stability and performance on harder tasks, whereas naive off-policy learning degrades in Ant Push and Ant Fall.
• Using raw state observations as goals provides immediate intrinsic reward signals and simple generalization across tasks.