[Paper Review] Distributed Distributional Deterministic Policy Gradients

Gabriel Barth-Maron, Matthew W. Hoffman+7 more|arXiv (Cornell University)|Apr 23, 2018

Reinforcement Learning in Robotics20 references283 citations

TL;DR

The paper introduces D4PG, a distributed off-policy actor-critic algorithm with a distributional critic and N-step returns, achieving state-of-the-art performance on diverse continuous control tasks.

ABSTRACT

This work adopts the very successful distributional perspective on reinforcement learning and adapts it to the continuous control setting. We combine this within a distributed framework for off-policy learning in order to develop what we call the Distributed Distributional Deep Deterministic Policy Gradient algorithm, D4PG. We also combine this technique with a number of additional, simple improvements such as the use of $N$-step returns and prioritized experience replay. Experimentally we examine the contribution of each of these individual components, and show how they interact, as well as their combined contributions. Our results show that across a wide variety of simple control tasks, difficult manipulation tasks, and a set of hard obstacle-based locomotion tasks the D4PG algorithm achieves state of the art performance.

Motivation & Objective

• Adopt a distributional perspective for the critic in continuous control settings.
• Develop a distributed, off-policy learning framework to accelerate data collection.
• Integrate N-step returns and prioritized experience replay to enhance learning.
• Systematically ablate components to understand their contributions and interactions.
• Demonstrate state-of-the-art performance across control, manipulation, and parkour tasks.

Proposed method

• Use a distributional critic (categorical distribution) to model return uncertainty.
• Extend DDPG with distributional Bellman updates and actor-critic gradients.
• Incorporate N-step returns into the distributional update.
• Distribute experience collection across K parallel actors writing to a shared replay table.
• Apply prioritized experience replay with importance sampling in a distributed setting.
• Leverage ApeX framework to manage parallel actors and replay-based learning.

Experimental results

Research questions

• RQ1 How does a distributional critic impact learning stability and performance in continuous control?
• RQ2 What are the effects of combining distributional updates with distributed actors, N-step returns, and prioritized replay?
• RQ3 Which components contribute most to performance gains across standard control, manipulation, and parkour tasks?
• RQ4 Is prioritization beneficial in the presence of distributional updates and distributed data collection?

Key findings

• Distributional updates improve performance, especially on harder tasks like humanoid and manipulation domains.
• N-step returns provide the largest relative gains among the proposed enhancements.
• The full D4PG algorithm achieves state-of-the-art performance across standard control, manipulation, and parkour tasks.
• Prioritized experience replay offers limited gains for D4PG and can sometimes be unnecessary.
• An unroll length of N=5 consistently outperforms N=1, with some instabilities for N=1 in certain tasks.
• Distributed actors plus a shared replay table substantially reduce wall-clock training time.