[Paper Review] URLB: Unsupervised Reinforcement Learning Benchmark

Michael Laskin, Denis Yarats+7 more|arXiv (Cornell University)|Oct 28, 2021

Reinforcement Learning in Robotics11 citations

TL;DR

URLB introduces a unified benchmark for unsupervised reinforcement learning (URL) that standardizes pre-training and fine-tuning across 12 continuous control tasks from the DeepMind Control Suite. It evaluates eight leading unsupervised RL algorithms using a shared optimization backbone and finds no existing method can fully solve the benchmark, highlighting the need for improved exploration and generalization in self-supervised RL.

ABSTRACT

Deep Reinforcement Learning (RL) has emerged as a powerful paradigm to solve a range of complex yet specific control tasks. Yet training generalist agents that can quickly adapt to new tasks remains an outstanding challenge. Recent advances in unsupervised RL have shown that pre-training RL agents with self-supervised intrinsic rewards can result in efficient adaptation. However, these algorithms have been hard to compare and develop due to the lack of a unified benchmark. To this end, we introduce the Unsupervised Reinforcement Learning Benchmark (URLB). URLB consists of two phases: reward-free pre-training and downstream task adaptation with extrinsic rewards. Building on the DeepMind Control Suite, we provide twelve continuous control tasks from three domains for evaluation and open-source code for eight leading unsupervised RL methods. We find that the implemented baselines make progress but are not able to solve URLB and propose directions for future research.

Motivation & Objective

• Address the lack of a unified benchmark for evaluating unsupervised reinforcement learning (URL) algorithms.
• Enable fair and transparent comparison of unsupervised RL methods by standardizing pre-training and fine-tuning procedures.
• Facilitate the development of generalist RL agents that can efficiently adapt to new tasks after unsupervised pre-training.
• Provide open-source code for both benchmark environments and eight leading unsupervised RL baselines with a shared optimization backbone.
• Identify key limitations in current URL methods and suggest promising research directions for future work.

Proposed method

• Design a two-phase benchmark: (i) reward-free pre-training using intrinsic rewards, and (ii) downstream fine-tuning with extrinsic rewards.
• Build on the DeepMind Control Suite to include three domains—Walker, Quadruped, and Jaco—across 12 continuous control tasks with varying difficulty.
• Implement eight leading unsupervised RL algorithms (e.g., RND, ProtoRL, DIAYN, SMM, APS) using a single, unified optimization backbone for fair comparison.
• Use pixel-based and state-based observations to evaluate generalization across input modalities.
• Standardize pre-training durations (100k to 2M frames) and fine-tuning protocols across all algorithms and tasks.
• Employ a fixed evaluation protocol with 10 random seeds per experiment to ensure reproducibility and statistical robustness.

Experimental results

Research questions

• RQ1 Can existing unsupervised RL algorithms achieve strong generalization and efficient few-shot adaptation on a standardized benchmark?
• RQ2 How do different intrinsic reward mechanisms (e.g., prediction error, entropy maximization, skill discovery) compare in terms of pre-training quality?
• RQ3 To what extent does the size of the skill space limit the performance of competence-based exploration methods in long-horizon, fixed-length environments?
• RQ4 Does the choice of observation modality (pixels vs. state) significantly affect the performance of unsupervised RL pre-training?
• RQ5 What are the key bottlenecks in current unsupervised RL methods that prevent them from solving complex, diverse control tasks?

Key findings

• No existing unsupervised RL algorithm can solve all 12 tasks in the URLB benchmark, indicating significant room for improvement in pre-training efficiency and exploration.
• Knowledge-based methods like RND and data-based methods like ProtoRL achieve comparable fine-tuning performance, suggesting both approaches are effective for diversity-driven exploration.
• Competence-based methods (e.g., DIAYN, SMM, APS) underperform significantly, likely due to limited skill space size, which leads to overfitting on low-level behaviors like falling or lying on the ground.
• The best-performing methods achieve average fine-tuning scores of around 300–500 in the Walker and Quadruped domains after 1M-frame pre-training, but fail to reach optimal performance on the most complex tasks.
• In the Jaco manipulation tasks, even top-performing methods achieve only modest success, with average scores below 50 in reach tasks, indicating poor generalization to dexterous manipulation.
• The benchmark reveals that current unsupervised RL methods struggle with long-horizon, fixed-length environments where extrinsic reward signals are absent during pre-training, especially when skill spaces are small.