[Paper Review] CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms

Shengyi Huang, Rousslan Fernand Julien Dossa+2 more|arXiv (Cornell University)|Nov 16, 2021

Reinforcement Learning in Robotics15 references86 citations

TL;DR

CleanRL provides high-quality, single-file implementations of deep RL algorithms with integrated experiment tracking and cloud-scale orchestration, aiding understanding, prototyping, and scalable research.

ABSTRACT

CleanRL is an open-source library that provides high-quality single-file implementations of Deep Reinforcement Learning algorithms. It provides a simpler yet scalable developing experience by having a straightforward codebase and integrating production tools to help interact and scale experiments. In CleanRL, we put all details of an algorithm into a single file, making these performance-relevant details easier to recognize. Additionally, an experiment tracking feature is available to help log metrics, hyperparameters, videos of an agent's gameplay, dependencies, and more to the cloud. Despite succinct implementations, we have also designed tools to help scale, at one point orchestrating experiments on more than 2000 machines simultaneously via Docker and cloud providers. Finally, we have ensured the quality of the implementations by benchmarking against a variety of environments. The source code of CleanRL can be found at https://github.com/vwxyzjn/cleanrl

Motivation & Objective

• Provide a transparent, easy-to-understand DRL implementation by consolidating algorithm details in single files.
• Enable rapid prototyping and debugging through a simplified code structure with global scope variables.
• Maintain competitive performance relative to reference implementations while keeping the code concise.
• Facilitate scalable experiments via integration with production tools and cloud infrastructure.
• Offer an open benchmark platform to track and compare RL experiments across domains.

Proposed method

• Deliver high-quality, self-contained single-file implementations for multiple DRL algorithms (e.g., PPO, DQN, C51, SAC, TD3) across diverse environments.
• Keep implementations concise (e.g., PPO at Atari in 337 LOC) to enhance readability and learning.
• Expose algorithms and environment specifics in a single file to make all learning-related details easily recognizable.
• Integrate with experiment tracking tools (Weights & Biases) and provide Tensorboard-based logging for metrics and videos.
• Provide cloud-ready workflows via Docker containers and Terraform for large-scale orchestration.
• Offer optional production-like tooling while remaining framework-agnostic to avoid hard dependencies.

Experimental results

Research questions

• RQ1 Does consolidating DRL algorithm implementations into single files improve transparency and ease of understanding compared to modular libraries?
• RQ2 How does the single-file design impact debugging speed and ease of prototyping new features in DRL research?
• RQ3 Can CleanRL-scale experiments effectively on cloud infrastructure and across thousands of runs while maintaining reproducibility?
• RQ4 How well do single-file implementations align with established benchmarks and competitive performance on standard environments?

Key findings

• Single-file PPO implementations achieve competitive performance with reference implementations (example: Breakout Atari) while using 337 LOC.
• Global-name-scope design enables easier interactive debugging and rapid prototyping by exposing most variables in the interactive shell.
• CleanRL supports extensive experiment tracking via Weights & Biases, recording code, dependencies, hyperparameters, metrics, and videos.
• The project demonstrates large-scale orchestration capabilities, having historically run experiments on more than 2000 machines simultaneously via Docker and cloud providers.
• Open RL Benchmark provides interactive access to track thousands of experiments across 7+ algorithms and multiple domains, including Atari and MuJoCo.
• The workflow has supported over 50,000+ hours of experiments using cloud-based orchestration in 2020.