[Paper Review] What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study

Marcin Andrychowicz, Anton Raichuk+10 more|arXiv (Cornell University)|Jun 10, 2020

Reinforcement Learning in Robotics26 references104 citations

TL;DR

The paper conducts a large-scale empirical study of over 50 design choices in a unified on-policy RL framework, training 250,000+ agents across five continuous control environments to identify practical recommendations.

ABSTRACT

In recent years, on-policy reinforcement learning (RL) has been successfully applied to many different continuous control tasks. While RL algorithms are often conceptually simple, their state-of-the-art implementations take numerous low- and high-level design decisions that strongly affect the performance of the resulting agents. Those choices are usually not extensively discussed in the literature, leading to discrepancy between published descriptions of algorithms and their implementations. This makes it hard to attribute progress in RL and slows down overall progress [Engstrom'20]. As a step towards filling that gap, we implement >50 such ``choices'' in a unified on-policy RL framework, allowing us to investigate their impact in a large-scale empirical study. We train over 250'000 agents in five continuous control environments of different complexity and provide insights and practical recommendations for on-policy training of RL agents.

Motivation & Objective

• Investigate how low- and high-level implementation choices impact on-policy RL performance.
• Quantify the effects of >50 configurable options within a unified PPO-based framework.
• Provide actionable guidelines for practitioners on hyperparameters, architectures, and training setups.

Proposed method

• Build a highly configurable on-policy agent based on PPO with exposed choices as configuration options.
• Train over 250,000 agents across five continuous control environments to assess performance.
• Evaluate each choice by analyzing the 95th percentile performance across groups of configurations and examining top-performing configurations.
• Use two-stage analysis to account for interactions between related choices and ensure robust conclusions.

Experimental results

Research questions

• RQ1 Which low- and high-level design choices most influence on-policy RL performance in continuous control tasks?
• RQ2 How do choices related to policy losses, network architectures, normalization, advantage estimation, training setup, timesteps handling, optimizers, and regularization affect learning speed and final performance?
• RQ3 What practical recommendations can be drawn for practitioners implementing on-policy RL in real-world settings?

Key findings

• PPO policy loss generally outperforms alternatives across most environments, with clipping threshold around 0.25 being a solid starting point.
• Separate value and policy networks often improve performance, with architecture width and activation choices (tanh) significantly impacting results; initial action distribution initialization has strong effects on training speed.
• Input normalization is crucial; value function normalization can dramatically influence performance depending on the environment; gradient clipping provides a modest boost.
• GAE and V-trace for advantage estimation outperform N-step; PPO-style value loss clipping and Huber loss are not broadly beneficial.
• Data collection strategy matters: going over experiences multiple times and recomputing advantages per data pass improves sample efficiency; more parallel environments can hurt some tasks but accelerate wall-clock time when resources permit.
• Adam optimizer with momentum and a tuned learning rate is a robust default; regularization via entropy or KL penalties offers limited or environment-dependent benefits.