[Paper Review] Benchmarking Deep Reinforcement Learning for Continuous Control

Yan Duan, Xi Chen+3 more|arXiv (Cornell University)|Apr 22, 2016

Reinforcement Learning in Robotics64 references966 citations

TL;DR

This paper introduces a comprehensive benchmark of 31 continuous control tasks and systematically evaluates a suite of deep reinforcement learning algorithms, highlighting TNPG, TRPO, and DDPG as effective on many tasks while noting challenges in hierarchical tasks.

ABSTRACT

Recently, researchers have made significant progress combining the advances in deep learning for learning feature representations with reinforcement learning. Some notable examples include training agents to play Atari games based on raw pixel data and to acquire advanced manipulation skills using raw sensory inputs. However, it has been difficult to quantify progress in the domain of continuous control due to the lack of a commonly adopted benchmark. In this work, we present a benchmark suite of continuous control tasks, including classic tasks like cart-pole swing-up, tasks with very high state and action dimensionality such as 3D humanoid locomotion, tasks with partial observations, and tasks with hierarchical structure. We report novel findings based on the systematic evaluation of a range of implemented reinforcement learning algorithms. Both the benchmark and reference implementations are released at https://github.com/rllab/rllab in order to facilitate experimental reproducibility and to encourage adoption by other researchers.

Motivation & Objective

• Motivate the need for a standardized, challenging continuous-control benchmark to quantify progress in deep RL.
• Provide a diverse suite of tasks spanning basic, locomotion, partial observability, and hierarchical structures implemented in physics simulators.
• Evaluate a range of gradient-based and gradient-free algorithms on deep neural policies to identify strengths and limitations.

Proposed method

• Define a finite-horizon discounted MDP and extend to POMDPs where needed.
• Implement 31 continuous control tasks across four categories using Box2D and MuJoCo simulators.
• Benchmark a collection of gradient-based (REINFORCE, TNPG, TRPO, RWR, REPS) and gradient-free (CEM, CMA-ES) methods, plus DDPG for online learning.
• Use neural policies (feed-forward for basic/locomotion/hierarchical tasks; recurrent for partially observable tasks) and standard baseline to reduce variance.
• Evaluate with multiple random seeds; grid-search hyperparameters for most algorithms and report mean performance with standard deviation.

Experimental results

Research questions

• RQ1 How do different reinforcement learning algorithms perform across a wide range of continuous control tasks?
• RQ2 What are the relative strengths and limitations of batch gradient-based methods versus online methods like DDPG in continuous control?
• RQ3 Do recurrent policies provide advantages in partially observable settings, and how do they interact with gradient-based versus gradient-free optimization?
• RQ4 Do hierarchical tasks reveal gaps in current algorithms and require new approaches to exploit structure?

Key findings

• TNPG and TRPO generally outperform other batch algorithms by providing stable learning through policy distribution constraints.
• REINFORCE can be effective on basic and locomotion tasks but may converge prematurely to local optima on some tasks.
• RWR can solve some basic tasks without hyperparameter tuning but struggles on locomotion tasks.
• DDPG offers faster convergence on some tasks (e.g., Half-Cheetah) but exhibits stability issues and reward scaling sensitivity.
• Most algorithms perform poorly on hierarchical tasks, indicating a need for methods that automatically discover and exploit hierarchical structure.
• CEM can excel on simple tasks but degrades with higher dimensionality and complex dynamics; CMA-ES can fail on high-dimensional tasks due to memory limits.