[Paper Review] Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation

Tejas D. Kulkarni, Karthik Narasimhan+2 more|arXiv (Cornell University)|Apr 20, 2016

Reinforcement Learning in Robotics48 references414 citations

TL;DR

The paper introduces h-DQN, a two-level deep Q-network framework that uses intrinsic goals and temporal abstraction to drive exploration, enabling learning in sparse-delayed reward tasks like Montezuma’s Revenge.

ABSTRACT

Learning goal-directed behavior in environments with sparse feedback is a major challenge for reinforcement learning algorithms. The primary difficulty arises due to insufficient exploration, resulting in an agent being unable to learn robust value functions. Intrinsically motivated agents can explore new behavior for its own sake rather than to directly solve problems. Such intrinsic behaviors could eventually help the agent solve tasks posed by the environment. We present hierarchical-DQN (h-DQN), a framework to integrate hierarchical value functions, operating at different temporal scales, with intrinsically motivated deep reinforcement learning. A top-level value function learns a policy over intrinsic goals, and a lower-level function learns a policy over atomic actions to satisfy the given goals. h-DQN allows for flexible goal specifications, such as functions over entities and relations. This provides an efficient space for exploration in complicated environments. We demonstrate the strength of our approach on two problems with very sparse, delayed feedback: (1) a complex discrete stochastic decision process, and (2) the classic ATARI game `Montezuma's Revenge'.

Motivation & Objective

• Address learning in environments with sparse and delayed rewards.
• Integrate temporal abstraction (options) with intrinsic motivation to guide exploration.
• Enable flexible goal specifications (entities and relations) to constrain exploration.
• Develop a scalable two-level deep RL architecture (meta-controller and controller) with joint training.

Proposed method

• Introduce h-DQN with a top-level meta-controller that selects intrinsic goals g and a lower-level controller that selects primitive actions a given s and g.
• Represent value functions as V(s,g) approximated by deep networks Q1(s,a;joint with g) and Q2(s,g) for the controller and meta-controller respectively.
• Train Q1 and Q2 with separate experience memories D1 and D2 and corresponding loss functions L1 and L2, following DQN-style updates.
• Use intrinsic rewards rt(g) provided by an internal critic when a goal g is reached, and extrinsic rewards ft from the environment for the meta-controller’s objective.
• Model transitions at two time scales: fast transitions (s,a,g,r,s') and slower, goal-oriented transitions (s,g, f, s') to learn at different temporal resolutions.
• Adopt epsilon-greedy exploration with adaptive annealing for both controllers, and update parameters via stochastic gradient descent on replay memories.

Experimental results

Research questions

• RQ1 Can hierarchical deep RL with intrinsic goals improve learning in sparse-reward environments compared to flat baselines?
• RQ2 Do goals defined over structured spaces like entities and relations improve exploration efficiency and data efficiency?
• RQ3 Does the two-level (controller and meta-controller) DQN framework scale to complex tasks like Montezuma’s Revenge?
• RQ4 What is the effect of learning and composing intrinsic goals on long-horizon credit assignment?

Key findings

• In a discrete stochastic decision process with delayed extrinsic rewards, the h-DQN approach found a policy that visits critical intermediate states, achieving significantly higher average rewards than standard Q-learning (around 0.13 vs 0.01 in the reported setup).
• On Montezuma’s Revenge, the two-phase training enabled the agent to learn to reach a key and subsequently open doors, yielding high extrinsic rewards where vanilla DQN and Gorila DQN perform poorly (basic DQN score reported as 0; Gorila DQN around 4.16 in the cited comparison).
• The architecture demonstrates that intrinsic motivation over entities/relations can effectively constrain exploration and support longer-horizon planning in visually rich, sparse-reward domains.
• The model shows progressive mastery of simpler goals before harder ones, indicating meaningful curriculum-like progression in goal selection during training.