[Paper Review] D4RL: Datasets for Deep Data-Driven Reinforcement Learning

Justin Fu, Aviral Kumar+3 more|arXiv (Cornell University)|Apr 15, 2020

Reinforcement Learning in Robotics50 references331 citations

TL;DR

D4RL introduces a comprehensive offline reinforcement learning benchmark suite with various tasks and datasets designed to reflect real-world data properties, accompanied by a standardized evaluation protocol and open-source implementations.

ABSTRACT

The offline reinforcement learning (RL) setting (also known as full batch RL), where a policy is learned from a static dataset, is compelling as progress enables RL methods to take advantage of large, previously-collected datasets, much like how the rise of large datasets has fueled results in supervised learning. However, existing online RL benchmarks are not tailored towards the offline setting and existing offline RL benchmarks are restricted to data generated by partially-trained agents, making progress in offline RL difficult to measure. In this work, we introduce benchmarks specifically designed for the offline setting, guided by key properties of datasets relevant to real-world applications of offline RL. With a focus on dataset collection, examples of such properties include: datasets generated via hand-designed controllers and human demonstrators, multitask datasets where an agent performs different tasks in the same environment, and datasets collected with mixtures of policies. By moving beyond simple benchmark tasks and data collected by partially-trained RL agents, we reveal important and unappreciated deficiencies of existing algorithms. To facilitate research, we have released our benchmark tasks and datasets with a comprehensive evaluation of existing algorithms, an evaluation protocol, and open-source examples. This serves as a common starting point for the community to identify shortcomings in existing offline RL methods and a collaborative route for progress in this emerging area.

Motivation & Objective

• Motivate offline reinforcement learning as a bridge between RL and supervised learning by leveraging large, previously-collected datasets.
• Design a benchmark that reflects real-world data collection challenges (e.g., narrow distributions, demonstrations, mixtures of policies).
• Provide a standardized evaluation protocol and open-source references to enable reproducible offline RL research.
• Cover a range of domains and difficulties to differentiate algorithm performance beyond simple tasks.

Proposed method

• Propose a suite of offline tasks and datasets across domains like Maze2D, AntMaze, Gym-MuJoCo, Adroit, FrankaKitchen, Flow, and CARLA to test offline RL in realistic settings.
• Characterize data collection properties such as non-Markovian behavior, sparse rewards, undirected data, and mixtures of policies to stress offline RL algorithms.
• Normalize performance via a consistent scoring scheme using random and expert baselines to enable cross-task comparison.
• Benchmark state-of-the-art offline RL algorithms and baselines on the proposed tasks to identify strengths and weaknesses.
• Release open-source APIs, datasets, and reference implementations to facilitate community adoption and reproducibility.

Experimental results

Research questions

• RQ1 How do current offline RL methods perform across a diverse set of datasets that reflect real-world data collection (e.g., demonstrations, mixtures of policies, non-Markovian data)?
• RQ2 What data properties most challenge offline RL algorithms, and which methods handle them best (e.g., conservative approaches vs. imitation-learning-based baselines)?
• RQ3 Do mixtures of policies or undirected data degrade offline RL performance, and can existing methods cope with such distributions?
• RQ4 Does offline RL offer advantages in sparse-reward or high-exploration domains compared to online baselines under fixed data?

Key findings

• Offline RL algorithms show varying performance across tasks, with larger gaps emerging on undirected and mixed-policy data.
• Conservative methods (e.g., BEAR, AWR, CQL, BCQ) handle biased and narrow data distributions well on Flow and Gym-MuJoCo domains.
• Tasks with non-Markovian data, stitching requirements, or high observational complexity (e.g., Maze2D, AntMaze, CARLA) remain challenging for most methods.
• In some domains with sparse rewards, offline RL methods can outperform online SAC, highlighting the potential to address exploration challenges offline.
• Datasets combining expert and suboptimal data reveal that many algorithms do not fully exploit mixed-quality data, underscoring the need for improved sample efficiency and data-driven regularization.
• The benchmark exposes deficiencies in existing algorithms beyond simple tasks, guiding future offline RL research.