[Paper Review] Deep Reinforcement Learning that Matters

Peter Henderson, Riashat Islam+4 more|arXiv (Cornell University)|Sep 19, 2017

Evolutionary Algorithms and ApplicationsComputer Science364 citations

TL;DR

This paper investigates reproducibility, experimental practices, and reporting in deep reinforcement learning, focusing on policy gradient methods, and proposes guidelines to improve rigor and comparability.

ABSTRACT

In recent years, significant progress has been made in solving challenging problems across various domains using deep reinforcement learning (RL). Reproducing existing work and accurately judging the improvements offered by novel methods is vital to sustaining this progress. Unfortunately, reproducing results for state-of-the-art deep RL methods is seldom straightforward. In particular, non-determinism in standard benchmark environments, combined with variance intrinsic to the methods, can make reported results tough to interpret. Without significance metrics and tighter standardization of experimental reporting, it is difficult to determine whether improvements over the prior state-of-the-art are meaningful. In this paper, we investigate challenges posed by reproducibility, proper experimental techniques, and reporting procedures. We illustrate the variability in reported metrics and results when comparing against common baselines and suggest guidelines to make future results in deep RL more reproducible. We aim to spur discussion about how to ensure continued progress in the field by minimizing wasted effort stemming from results that are non-reproducible and easily misinterpreted.

Motivation & Objective

• Assess sources of variability in reproducibility for deep RL experiments.
• Evaluate how hyperparameters, architecture, seeds, and environments affect results.
• Evaluate the impact of different codebases and implementation details on baselines.
• Propose guidelines and statistical practices to improve reproducibility and fair comparisons.

Proposed method

• Review and experimentally analyze factors affecting reproducibility in policy gradient methods for continuous control.
• Conduct controlled experiments varying hyperparameters, network architectures, reward scaling, seeds, and environments.
• Compare multiple baseline implementations (e.g., OpenAI Baselines, TRPO, PPO, DDPG, ACKTR) across MuJoCo tasks.
• Use mean and standard error over multiple seeds; discuss significance testing and bootstrap methods.

Experimental results

Research questions

• RQ1 How do hyperparameters influence baseline performance across algorithms and environments?
• RQ2 What is the impact of network architecture and activation choices on learning outcomes?
• RQ3 How do random seeds, number of trials, and environment stochasticity affect reported results?
• RQ4 To what extent do different codebases alter baseline performance?

Key findings

• Hyperparameters can have large and inconsistent effects across algorithms and environments.
• Network architecture and activation functions substantially influence performance and interact with the chosen algorithm.
• Random seeds and the number of trials can cause large performance variance; averaging over seeds without proper statistical framing can be misleading.
• Environment properties (stability vs. instability) strongly affect algorithm performance and can change which method performs best.
• Implementation details across codebases can produce substantial performance differences, highlighting the need for reporting all details and sharing code.
• Significance testing and bootstrap analyses provide meaningful insights into whether observed gains are reliable.