[Paper Review] Active Hierarchical Exploration with Stable Subgoal Representation Learning

Siyuan Li, Jin Zhang+3 more|arXiv (Cornell University)|May 31, 2021

Mobile Crowdsensing and Crowdsourcing7 citations

TL;DR

This paper proposes HESS, a novel active hierarchical exploration method that stabilizes subgoal representation learning via a state-specific regularization, enabling efficient and stable high-level policy learning in long-horizon, sparse-reward continuous control tasks. By combining a regularized novelty measure with a reachability-based potential function, HESS actively selects promising subgoals without intrinsic rewards, achieving state-of-the-art sample efficiency and performance on challenging sparse-reward benchmarks.

ABSTRACT

Goal-conditioned hierarchical reinforcement learning (GCHRL) provides a promising approach to solving long-horizon tasks. Recently, its success has been extended to more general settings by concurrently learning hierarchical policies and subgoal representations. Although GCHRL possesses superior exploration ability by decomposing tasks via subgoals, existing GCHRL methods struggle in temporally extended tasks with sparse external rewards, since the high-level policy learning relies on external rewards. As the high-level policy selects subgoals in an online learned representation space, the dynamic change of the subgoal space severely hinders effective high-level exploration. In this paper, we propose a novel regularization that contributes to both stable and efficient subgoal representation learning. Building upon the stable representation, we design measures of novelty and potential for subgoals, and develop an active hierarchical exploration strategy that seeks out new promising subgoals and states without intrinsic rewards. Experimental results show that our approach significantly outperforms state-of-the-art baselines in continuous control tasks with sparse rewards.

Motivation & Objective

• To address the instability of online-learned subgoal representations in goal-conditioned hierarchical reinforcement learning (GCHRL), which hinders effective high-level exploration in long-horizon, sparse-reward tasks.
• To improve exploration efficiency by designing a stable subgoal representation space that supports reliable novelty and potential estimation.
• To develop an active exploration strategy that selects high-potential, novel subgoals without relying on intrinsic rewards, enabling direct and efficient policy learning.
• To achieve superior sample efficiency and performance in challenging continuous control environments with sparse external rewards.

Proposed method

• Introduces a state-specific regularization that stabilizes subgoal representations by constraining updates to embeddings that already satisfy the representation objective, improving learning stability and efficiency.
• Employs a contrastive objective for subgoal representation learning, inspired by prior work, to learn meaningful and disentangled subgoal embeddings in latent space.
• Designs a regularized novelty measure based on visit counts in the subgoal space, enhanced by a potential function that evaluates the reachability of neighboring regions to prioritize exploratory subgoals.
• Uses prioritized sampling (Hinton, 2007) to focus training on samples with higher losses, improving representation learning efficiency without sacrificing stability.
• Develops an active exploration strategy that directly selects subgoals based on combined novelty and potential, bypassing the need for intrinsic reward maximization.
• Applies the high-level policy to select subgoals in the latent space using the regularized novelty and potential scores, enabling targeted exploration of unvisited and reachable regions.

Experimental results

Research questions

• RQ1 Can a regularization mechanism stabilize online-learned subgoal representations in hierarchical reinforcement learning, thereby improving exploration stability?
• RQ2 How can novelty and potential be effectively measured in a dynamically changing subgoal space to guide high-level exploration?
• RQ3 Can active subgoal selection based on regularized novelty and potential outperform reactive exploration with intrinsic rewards in sparse-reward environments?
• RQ4 To what extent does the proposed method improve sample efficiency and performance in long-horizon continuous control tasks?

Key findings

• HESS significantly outperforms state-of-the-art baselines in continuous control tasks with sparse rewards, demonstrating superior sample efficiency and final performance.
• The ablation study confirms that the potential measure is critical: without it, subgoals concentrate on low-utility regions (e.g., corners), leading to poor success rates.
• The method remains robust across hyper-parameters, with stable performance over a wide range of values for scaling factor α, extended distance de, and stable ratio k%, indicating generalization and reliability.
• The proposed stability regularization improves representation learning efficiency without compromising stability, enabling effective prioritized sampling.
• The active exploration strategy achieves better performance than reactive methods with intrinsic rewards, as it directly targets promising subgoals without delay in intrinsic reward learning.
• The method achieves strong results across diverse tasks, including Ant Maze and Ant Push (Images), with a single set of hyperparameters, demonstrating generalization and robustness.