[Paper Review] Continuous control with deep reinforcement learning

Timothy Lillicrap, Jonathan J. Hunt+6 more|arXiv (Cornell University)|Sep 9, 2015

Reinforcement Learning in Robotics26 references5,361 citations

TL;DR

The paper introduces Deep Deterministic Policy Gradient (DDPG), an off-policy actor-critic algorithm for continuous control using deep networks, with replay buffers and target networks to enable stable learning from both low-dimensional states and raw pixel inputs across many tasks.

ABSTRACT

We adapt the ideas underlying the success of Deep Q-Learning to the continuous action domain. We present an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. Using the same learning algorithm, network architecture and hyper-parameters, our algorithm robustly solves more than 20 simulated physics tasks, including classic problems such as cartpole swing-up, dexterous manipulation, legged locomotion and car driving. Our algorithm is able to find policies whose performance is competitive with those found by a planning algorithm with full access to the dynamics of the domain and its derivatives. We further demonstrate that for many of the tasks the algorithm can learn policies end-to-end: directly from raw pixel inputs.

Motivation & Objective

• Address the challenge of scalable reinforcement learning in continuous action spaces.
• Develop a model-free, off-policy algorithm that leverages deep function approximators.
• Stabilize learning with replay buffers and target networks inspired by DQN.
• Demonstrate end-to-end learning from raw pixel inputs in complex control tasks.

Proposed method

• Adopt an actor-critic framework with a deterministic policy gradient (DPG) to handle continuous actions.
• Use deep neural networks to represent both actor μ(s|θ^μ) and critic Q(s,a|θ^Q).
• Incorporate a replay buffer to break sample correlations and enable minibatch updates.
• Introduce soft target networks for both actor and critic to stabilize learning (θ^Q' and θ^μ').
• Apply batch normalization to inputs and hidden layers to handle heterogeneous state scales.
• Add exploration noise to the actor via an Ornstein–Uhlenbeck process for temporally correlated exploration.
• Train with off-policy updates using a target y_t = r_t + γ Q'(s_{t+1}, μ'(s_{t+1})|θ^{Q'}) junction, updating parameters with gradient-based rules (policy gradient and critic loss).

Experimental results

Research questions

• RQ1 Can a deterministic policy gradient-based actor-critic method scale to high-dimensional continuous action spaces using deep function approximators?
• RQ2 Do replay buffers and target networks enable stable, off-policy learning for continuous control from both low-dimensional and pixel-based observations?
• RQ3 Is end-to-end learning from raw pixels feasible for complex control tasks with DDPG?
• RQ4 How does DDPG compare to planning-based methods when dynamics and derivatives are known?
• RQ5 What is the impact of components like batch normalization and target networks on performance and stability?

Key findings

• DDPG with replay buffer and soft target networks achieves competitive performance across 20+ continuous-control tasks.
• In several tasks, DDPG policies surpass the performance of a planner with full knowledge of dynamics and derivatives.
• Learning from pixels is feasible for many tasks, achieving similar or superior results to low-dimensional inputs under the same hyper-parameters.
• Target networks (both for actor and critic) are crucial for stability and performance; removing them degrades results.
• Batch normalization aids learning across tasks with differing state scales, enabling broader generalization.
• The algorithm can learn end-to-end from pixels in many environments, with action repeats aiding observability in high-dimensional settings.