[Paper Review] Efficient Hierarchical Exploration with Stable Subgoal Representation Learning.

Siyuan Li, Jin Zhang+2 more|arXiv (Cornell University)|May 31, 2021

Reinforcement Learning in Robotics47 references4 citations

TL;DR

This paper proposes a stable subgoal representation learning method for hierarchical reinforcement learning that uses state-specific regularization to stabilize embeddings in well-explored regions while enabling updates in unexplored areas. By leveraging stable representations, the method introduces novelty and potential-based subgoal selection, significantly improving exploration efficiency and performance on sparse-reward continuous control tasks compared to state-of-the-art baselines.

ABSTRACT

Goal-conditioned hierarchical reinforcement learning (HRL) serves as a successful approach to solving complex and temporally extended tasks. Recently, its success has been extended to more general settings by concurrently learning hierarchical policies and subgoal representations. However, online subgoal representation learning exacerbates the non-stationary issue of HRL and introduces challenges for exploration in high-level policy learning. In this paper, we propose a state-specific regularization that stabilizes subgoal embeddings in well-explored areas while allowing representation updates in less explored state regions. Benefiting from this stable representation, we design measures of novelty and potential for subgoals, and develop an efficient hierarchical exploration strategy that actively seeks out new promising subgoals and states. Experimental results show that our method significantly outperforms state-of-the-art baselines in continuous control tasks with sparse rewards and further demonstrate the stability and efficiency of the subgoal representation learning of this work, which promotes superior policy learning.

Motivation & Objective

• To address the non-stationarity and exploration challenges in online subgoal representation learning within hierarchical reinforcement learning.
• To stabilize subgoal embeddings in well-visited state regions while allowing dynamic updates in underexplored areas.
• To develop an efficient hierarchical exploration strategy that actively identifies new, promising subgoals using representation stability.
• To improve policy learning performance in complex, temporally extended tasks with sparse rewards.

Proposed method

• Introduces a state-specific regularization technique that constrains subgoal representation updates based on state visitation frequency.
• Uses the regularized subgoal representations to compute subgoal novelty and potential, guiding exploration toward unexplored yet promising regions.
• Employs a hierarchical RL framework where the high-level policy selects subgoals based on their novelty and potential, while the low-level policy executes actions to reach them.
• Maintains stable representations in frequently visited states to reduce distributional shift and improve training stability.
• Leverages the stability of subgoal representations to define meaningful intrinsic rewards for subgoal selection.
• Designs a novel exploration strategy that prioritizes subgoals with high potential and high novelty, promoting efficient coverage of the state space.

Experimental results

Research questions

• RQ1 How can subgoal representation learning be stabilized during online training in hierarchical reinforcement learning?
• RQ2 What role does representation stability play in improving exploration efficiency in high-dimensional, sparse-reward environments?
• RQ3 Can novelty and potential metrics based on stable subgoal representations effectively guide hierarchical exploration?
• RQ4 How does the proposed method compare to state-of-the-art approaches in terms of sample efficiency and final performance on complex control tasks?

Key findings

• The proposed method achieves significant performance gains over state-of-the-art baselines on continuous control tasks with sparse rewards.
• Stable subgoal representation learning reduces distributional shift and improves training stability during online learning.
• The novelty and potential-based subgoal selection strategy enables more efficient exploration, covering a larger portion of the state space.
• The method demonstrates superior sample efficiency, achieving higher returns with fewer environment interactions.
• The ablation study confirms that state-specific regularization is essential for maintaining stable representations and improving final policy performance.
• Empirical results show that stable representations lead to more reliable and effective subgoal discovery, enhancing overall policy learning.