[Paper Review] Dopamine: A Research Framework for Deep Reinforcement Learning

Pablo Samuel Castro, Subhodeep Moitra+3 more|arXiv (Cornell University)|Dec 14, 2018

Reinforcement Learning in RoboticsComputer Science6 references172 citations

TL;DR

Dopamine is a small, open-source TensorFlow framework focused on stable, reproducible value-based deep RL research in the Arcade Learning Environment, emphasizing simplicity to support algorithmic research and instructional use. It provides four agents, baseline baselines, and reproducible experiment tooling.

ABSTRACT

Deep reinforcement learning (deep RL) research has grown significantly in recent years. A number of software offerings now exist that provide stable, comprehensive implementations for benchmarking. At the same time, recent deep RL research has become more diverse in its goals. In this paper we introduce Dopamine, a new research framework for deep RL that aims to support some of that diversity. Dopamine is open-source, TensorFlow-based, and provides compact and reliable implementations of some state-of-the-art deep RL agents. We complement this offering with a taxonomy of the different research objectives in deep RL research. While by no means exhaustive, our analysis highlights the heterogeneity of research in the field, and the value of frameworks such as ours.

Motivation & Objective

• Characterize diverse research objectives in deep RL and identify software needs for each objective.
• Present a compact, self-contained framework designed to support algorithmic research and instructional use in deep RL.
• Provide reproducible baselines and tooling to compare agents consistently across ALE.
• Offer interactive notebooks, pretrained models, and training data to facilitate replication and learning.

Proposed method

• Introduce Dopamine as a 12-file, TensorFlow-based framework focusing on compactness and reliability.
• Implement and provide four established value-based agents (DQN, C51, Rainbow-like, IQN) with ALE compatibility.
• Use gin-config for centralized experiment configuration to ensure reproducibility and easy parameter swapping.
• Provide extensive tests (coverage >98%) and ready-made baselines for consistent benchmarking.
• Offer Colab notebooks, pretrained baselines, and data to facilitate replication and visualization.
• Discuss design decisions through a taxonomy of research objectives (architecture, comprehensive studies, visualization, algorithmic research, instructional).],
• research_questions

Experimental results

Research questions

• RQ1 How do different research objectives in deep RL map to software needs and framework design?
• RQ2 Can a compact, self-contained framework support meaningful algorithmic research and instructional use without sacrificing reproducibility?
• RQ3 What are the effects of environment and evaluation choices (e.g., ALE settings, sticky actions) on reported RL performance?
• RQ4 How can a common baseline set enable fair comparisons across ALE-based value-based agents?
• RQ5 What infrastructure and tooling (notebooks, checkpoints, logs) best support reproducible deep RL research?

Key findings

• Dopamine demonstrates that a compact (12 files, ~2000 lines of Python) framework can implement multiple state-of-the-art value-based agents for ALE.
• A gin-config driven setup enables reproducible experiments with centralized, easily adjustable hyperparameters and full baselines for fair benchmarking.
• Default settings (sticky actions, termination criteria, and standardized hyperparameters) materially affect learning dynamics and reported performance across agents, highlighting reproducibility concerns identified in prior work.
• The framework provides comprehensive tests (>98% coverage) and ready-to-use baselines (including pre-trained checkpoints and TensorBoard logs) to facilitate replication and comparison.
• Dopamine emphasizes algorithmic research and instructional use, showing that simplicity and reliability can coexist with useful research tooling and educational resources.
• Baseline comparisons in ALE reveal that default settings can change agent dominance (e.g., C51 vs Rainbow vs IQN) depending on configuration; the framework exposes these effects clearly.