[Paper Review] Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning

Junsu Kim, Younggyo Seo+1 more|arXiv (Cornell University)|Oct 26, 2021

Reinforcement Learning in Robotics52 references19 citations

TL;DR

This paper proposes HIGL, a hierarchical reinforcement learning framework that improves exploration efficiency by guiding subgoal generation through landmarks—promising, informative states selected via coverage and novelty criteria. By constructing a landmark-based graph and selecting the nearest landmark on the shortest path to the goal, HIGL reduces the high-level policy's action space, achieving superior performance on long-horizon control tasks, including a 65.1% success rate in Ant Maze (sparse), outperforming HRAC's 17.6%.

ABSTRACT

Goal-conditioned hierarchical reinforcement learning (HRL) has shown promising results for solving complex and long-horizon RL tasks. However, the action space of high-level policy in the goal-conditioned HRL is often large, so it results in poor exploration, leading to inefficiency in training. In this paper, we present HIerarchical reinforcement learning Guided by Landmarks (HIGL), a novel framework for training a high-level policy with a reduced action space guided by landmarks, i.e., promising states to explore. The key component of HIGL is twofold: (a) sampling landmarks that are informative for exploration and (b) encouraging the high-level policy to generate a subgoal towards a selected landmark. For (a), we consider two criteria: coverage of the entire visited state space (i.e., dispersion of states) and novelty of states (i.e., prediction error of a state). For (b), we select a landmark as the very first landmark in the shortest path in a graph whose nodes are landmarks. Our experiments demonstrate that our framework outperforms prior-arts across a variety of control tasks, thanks to efficient exploration guided by landmarks.

Motivation & Objective

• To address the challenge of poor exploration in goal-conditioned hierarchical reinforcement learning due to large high-level action spaces.
• To improve sample efficiency and training performance in long-horizon continuous control tasks with sparse rewards.
• To develop a method that reduces the high-level policy's action space by focusing on promising, informative states (landmarks).
• To guide subgoal generation toward landmarks that are both diverse (coverage) and novel (prediction error), enhancing exploration.
• To enable scalable and effective subgoal discovery without relying on domain-specific knowledge or pre-defined subgoal spaces.

Proposed method

• Landmarks are sampled using two criteria: (a) coverage-based sampling to maximize spatial dispersion across visited states, and (b) novelty-based sampling using prediction error of a state autoencoder to identify rare or unexplored states.
• A landmark graph is constructed with nodes representing sampled landmarks, the current state, and the goal, enabling shortest-path planning to identify the most urgent landmark.
• The high-level policy is trained to generate subgoals directed toward the selected landmark, effectively narrowing the action space to reachable and promising directions.
• The landmark selection process uses Dijkstra’s algorithm on the landmark graph to find the first landmark on the shortest path from the current state to the goal.
• The framework integrates with existing goal-conditioned HRL architectures, using the same low-level policy and reward shaping, while modifying only the high-level policy’s action space and training objective.
• The method operates during training only, with no planning overhead at inference time, preserving deployment efficiency.

Experimental results

Research questions

• RQ1 Can landmark-based subgoal guidance reduce the high-level policy’s action space while improving exploration efficiency in long-horizon tasks?
• RQ2 How do coverage-based and novelty-based landmark sampling schemes compare in promoting diverse and effective exploration?
• RQ3 Does selecting the nearest landmark on the shortest path to the goal lead to faster convergence and higher success rates than uniform sampling or k-adjacency region methods?
• RQ4 To what extent does HIGL improve performance in environments with sparse rewards, where exploration is particularly challenging?
• RQ5 Can landmark-guided subgoal generation be effectively scaled to high-dimensional state spaces, such as image-based observations?

Key findings

• HIGL achieves a 65.1% success rate in the Ant Maze (sparse) environment, significantly outperforming HRAC’s 17.6% under the same conditions.
• In the Ant Maze (U-shape, dense) environment, HIGL demonstrates superior sample efficiency, with coverage-based landmarks spreading across visited regions and novelty-based landmarks concentrating at the frontier of exploration.
• The combination of coverage and novelty-based landmark sampling results in more effective exploration, as evidenced by broader state space coverage and faster convergence.
• HIGL reduces the high-level policy’s action space by focusing on promising, reachable landmarks, leading to more directed and efficient subgoal generation.
• Despite a 13-hour training time for 1M timesteps (vs. 6 hours for HRAC), HIGL’s planning cost is justified by improved sample efficiency and performance, especially in sparse-reward settings.
• The framework shows strong generalization across multiple MuJoCo-based long-horizon control tasks, confirming its robustness and scalability.