[Paper Review] RLlib: Abstractions for Distributed Reinforcement Learning

Eric Liang, Richard Liaw+7 more|arXiv (Cornell University)|Dec 26, 2017

Reinforcement Learning in RoboticsComputer Science328 citations

TL;DR

RLlib presents a hierarchical, logically centralized control model for distributed reinforcement learning and provides scalable abstractions and primitives via Ray to implement a wide range of RL algorithms with high performance and code reuse.

ABSTRACT

Reinforcement learning (RL) algorithms involve the deep nesting of highly irregular computation patterns, each of which typically exhibits opportunities for distributed computation. We argue for distributing RL components in a composable way by adapting algorithms for top-down hierarchical control, thereby encapsulating parallelism and resource requirements within short-running compute tasks. We demonstrate the benefits of this principle through RLlib: a library that provides scalable software primitives for RL. These primitives enable a broad range of algorithms to be implemented with high performance, scalability, and substantial code reuse. RLlib is available at https://rllib.io/.

Motivation & Objective

• Motivate the need for composable, scalable primitives in distributed reinforcement learning.
• Propose a hierarchical, logically centralized control model to encapsulate parallelism within RL components.
• Introduce RLlib as a library that implements these abstractions to enable rapid development of scalable RL algorithms.
• Demonstrate performance and scalability of RLlib across multiple RL workloads and baselines.

Proposed method

• Propose hierarchical and logically centralized control to manage RL components and nested parallelism.
• Define a policy graph abstraction where a policy pi, postprocessor rho, loss L, and utilities u are specified by the algorithm.
• Implement PolicyEvaluator to collect experiences and PolicyOptimizer to perform distributed updates over evaluator replicas.
• Build RLlib on Ray to enable hierarchical task scheduling with actors and remote tasks.
• Show multiple RL algorithm implementations (DQNs, PPO, A3C, DDPG, ES, AlphaGo-like setups) within the same abstractions.
• Evaluate performance against specialized systems and demonstrate scalability metrics.

Experimental results

Research questions

• RQ1 Can a logically centralized, hierarchical control model simplify and generalize the composition of distributed RL components?
• RQ2 Does RLlib’s abstraction enable scalable performance across a broad family of RL algorithms and hardware configurations?
• RQ3 How do different policy optimizers (e.g., asynchronous, sharded parameter servers, local multi-GPU) compare in throughput and scalability?
• RQ4 What is the impact of nested, distributed task structures on data transfer, scheduling, and fault tolerance in RL workloads?

Key findings

• RLlib achieves state-of-the-art performance across a broad range of RL workloads using hierarchical control and short-running tasks.
• Ape-X within RLlib scales to 160k environment frames per second with 256 workers, outperforming a reference setup.
• Single-node Pong (PPO) and Pendulum benchmarks demonstrate high throughput and efficient scaling on CPU and GPU resources.
• Evolution Strategies scale to 8192 cores on Humanoid-v1, achieving a median time of 3.7 minutes for reward 6000, more than twice the best published result.
• RLlib’s local multi-GPU policy optimizer can outperform distributed allreduce in certain configurations, due to better GPU memory locality.
• RLlib matches or exceeds the performance of specialized systems for PPO and A3C benchmarks, using the same hyperparameters.