[Paper Review] Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control

Riashat Islam, Peter Henderson+2 more|arXiv (Cornell University)|Aug 10, 2017

Reinforcement Learning in RoboticsComputer Science12 references185 citations

TL;DR

The paper studies how hyper-parameter tuning, environment stochasticity, and random seeds affect reproducibility of DDPG and TRPO on Hopper-v1 and Half-Cheetah-v1, and provides guidelines for fair baselines and reporting.

ABSTRACT

Policy gradient methods in reinforcement learning have become increasingly prevalent for state-of-the-art performance in continuous control tasks. Novel methods typically benchmark against a few key algorithms such as deep deterministic policy gradients and trust region policy optimization. As such, it is important to present and use consistent baselines experiments. However, this can be difficult due to general variance in the algorithms, hyper-parameter tuning, and environment stochasticity. We investigate and discuss: the significance of hyper-parameters in policy gradients for continuous control, general variance in the algorithms, and reproducibility of reported results. We provide guidelines on reporting novel results as comparisons against baseline methods such that future researchers can make informed decisions when investigating novel methods.

Motivation & Objective

• Assess sources of variance in policy gradient methods for continuous control.
• Evaluate sensitivity to hyper-parameters across TRPO and DDPG on Hopper and Half-Cheetah.
• Quantify how hyper-parameters and random seeds influence reproducibility of results.
• Propose guidelines for reporting baselines and experimental procedures in continuous control RL.

Proposed method

• Use MuJoCo/OpenAI Gym environments Hopper-v1 and Half-Cheetah-v1 as testbeds.
• Reproduce TRPO and DDPG using implementations from prior works.
• Vary hyper-parameters such as policy network architecture, batch size, step size, regularization, lambda for GAE, reward scale, and learning rates.
• Run 5000 iterations with 5 random seeds and analyze results with multiple trials.
• Assess variance by comparing results across different seeds and configurations.
• Compare results with prior baselines and report multiple metrics including average return and standard deviation.
• Investigate the impact of environment stochasticity on reproducibility through repeated trials.

Experimental results

Research questions

• RQ1 How do hyper-parameters influence the performance and reproducibility of TRPO and DDPG on Hopper and Half-Cheetah?
• RQ2 What is the role of random seeds and environment stochasticity in the variability of reported results?
• RQ3 Which reporting metrics and experimental protocols are necessary to enable fair baselines for continuous control RL methods?
• RQ4 Do baseline results from different papers remain consistent under careful tuning and thorough averaging across trials?

Key findings

• Hyper-parameters and stochasticity cause large performance variance in both TRPO and DDPG.
• Network architecture has a substantial impact on Half-Cheetah, less so on Hopper; DDPG is particularly unstable on Hopper.
• Larger TRPO batch sizes improve performance more than smaller ones; DDPG shows limited gains from batch size changes.
• Reward scaling and actor/critic learning rates have environment-dependent effects, with inconsistent results across Half-Cheetah and Hopper.
• Even with tuned hyper-parameters, results vary significantly across random seeds, underscoring the need to average over many trials.
• Many prior works report only select metrics, which can misrepresent baselines; comprehensive reporting of all metrics and hyper-parameters is advised.