[Paper Review] d3rlpy: An Offline Deep Reinforcement Learning Library

Takuma Seno, Michita Imai|arXiv (Cornell University)|Nov 6, 2021

Reinforcement Learning in Robotics35 references42 citations

TL;DR

d3rlpy is an open-source offline and online deep RL library in Python with a plug-and-play API, standardized interfaces, and a large-scale reproducibility benchmark using D4RL and Atari datasets.

ABSTRACT

In this paper, we introduce d3rlpy, an open-sourced offline deep reinforcement learning (RL) library for Python. d3rlpy supports a set of offline deep RL algorithms as well as off-policy online algorithms via a fully documented plug-and-play API. To address a reproducibility issue, we conduct a large-scale benchmark with D4RL and Atari 2600 dataset to ensure implementation quality and provide experimental scripts and full tables of results. The d3rlpy source code can be found on GitHub: \url{https://github.com/takuseno/d3rlpy}.

Motivation & Objective

• Provide an open-source, fully documented offline deep RL library in Python.
• Offer a plug-and-play API for both offline and online RL algorithms built with PyTorch.
• Standardize interfaces to ease integration and reproducibility across researchers.
• Demonstrate reproducibility by benchmarking against D4RL and Atari 2600 datasets and providing experimental scripts.

Proposed method

• Introduce a scikit-learn-styled API for offline and online RL training (fit and fit_online).
• Define a dedicated MDPDataset component for standardized offline RL data handling.
• Implement a hierarchical Algorithm design with AlgorithmImpl to separate high-level training flow from low-level updates.
• Provide EncoderFactory and QFunctionFactory for customizable neural architectures and distributional Q-functions.
• Support data preprocessing with Scaler, ActionScaler, and RewardScaler for stable offline training.
• Benchmark algorithms using faithful implementations on D4RL and Atari datasets with full reproducibility scripts.

Experimental results

Research questions

• RQ1 How can offline deep RL algorithms be standardized into a plug-and-play library?
• RQ2 What design choices yield reproducible and faithful benchmarks for offline RL across datasets like D4RL and Atari?
• RQ3 Can a unified API support both offline-only and offline-to-online training workflows effectively?
• RQ4 What preprocessing and model customization options improve offline RL performance and reproducibility?

Key findings

• d3rlpy provides a range of offline and online RL algorithms with a fully documented API and plug-and-play compatibility.
• The library is built around standardized data handling (MDPDataset) and modular components (EncoderFactory, QFunctionFactory) to support customization.
• A large-scale benchmark against D4RL and Atari datasets demonstrates faithful implementation quality and reproducibility, with scripts and full results available.
• Comprehensive preprocessing (Scaler, ActionScaler, RewardScaler) and flexible model architectures aid offline RL performance.
• The design enables seamless transitions from offline training to online fine-tuning, facilitating further research in policy improvement.