[Paper Review] Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning

Sainbayar Sukhbaatar, Emily Denton+2 more|arXiv (Cornell University)|Nov 22, 2018

Reinforcement Learning in Robotics18 references15 citations

TL;DR

This paper proposes an unsupervised self-play framework to learn continuous goal embeddings and low-level policies for hierarchical reinforcement learning, enabling high-level policies to direct agents via sub-goal vectors. The method achieves strong performance on complex tasks like AntGather, outperforming non-hierarchical baselines by learning a disentangled, spatially meaningful goal representation through adversarial goal generation.

ABSTRACT

In hierarchical reinforcement learning a major challenge is determining appropriate low-level policies. We propose an unsupervised learning scheme, based on asymmetric self-play from Sukhbaatar et al. (2018), that automatically learns a good representation of sub-goals in the environment and a low-level policy that can execute them. A high-level policy can then direct the lower one by generating a sequence of continuous sub-goal vectors. We evaluate our model using Mazebase and Mujoco environments, including the challenging AntGather task. Visualizations of the sub-goal embeddings reveal a logical decomposition of tasks within the environment. Quantitatively, our approach obtains compelling performance gains over non-hierarchical approaches.

Motivation & Objective

• To address the challenge of learning effective, general-purpose low-level policies and sub-goal representations in hierarchical reinforcement learning without explicit supervision.
• To enable high-level policies to direct agents using continuous goal vectors that capture controllable environmental dynamics.
• To improve sample efficiency and performance on sparse-reward tasks by pre-training the low-level policy via unsupervised asymmetric self-play.
• To learn a goal embedding space that abstracts irrelevant environmental details while preserving discriminability and achievability of sub-goals.

Proposed method

• Pre-train a low-level actor (Bob) using asymmetric self-play, where Bob generates and attempts to solve self-imposed tasks via goal vectors.
• Use a goal-conditional policy network that takes both the current state and a goal vector as input to guide low-level behavior.
• Leverage an adversarial reward structure where a second agent (Alice) proposes diverse, challenging tasks to prevent policy collapse and encourage exploration.
• Train the high-level policy (Charlie) using sparse task rewards, with each action corresponding to a 50-step horizon and a sequence of goal vectors.
• Use a distance-based success metric (D) during self-play that focuses on controllable state dimensions (e.g., x-y position in Ant), abstracting out irrelevant variables like height or joint angles.
• Apply entropy regularization to the goal proposal policy to maintain diversity in the goal space and avoid degenerate solutions.

Experimental results

Research questions

• RQ1 Can unsupervised self-play effectively learn a general-purpose, continuous goal embedding space that supports diverse and controllable low-level behaviors?
• RQ2 How does pre-training via self-play improve the sample efficiency and performance of hierarchical RL on sparse-reward environments?
• RQ3 To what extent does the learned goal embedding space reflect meaningful, disentangled, and spatially coherent sub-goals in the environment?
• RQ4 Can the high-level policy generalize to complex, long-horizon tasks by composing learned sub-goals, even when trained with only sparse task-level rewards?

Key findings

• The proposed model achieved an average return of 0.5 on the AntGather task, while non-hierarchical baselines like PPO, REINFORCE, and self-play failed to learn any positive reward.
• Visualizations of goal embeddings revealed a spatially coherent structure that mirrored the actual XY locations of proposed goals, indicating effective learning of controllable environmental dynamics.
• Bob’s policy generalized to unseen goal vectors, covering significantly longer distances (6+ steps) than during self-play (max 0.7 steps), demonstrating robust generalization.
• The diversity of proposed goals increased over time, with Alice learning to generate tasks across all directions, aided by entropy regularization.
• Trajectories for fixed goal vectors showed consistent directional movement toward target locations, confirming that the goal embedding enables precise control.
• The goal embedding space abstracted away irrelevant state dimensions (e.g., height, joint angles), focusing only on controllable aspects of the environment.