[Paper Review] Ray: A Distributed Framework for Emerging AI Applications

Philipp Moritz, Robert Nishihara+9 more|arXiv (Cornell University)|Dec 16, 2017

Reinforcement Learning in Robotics53 references253 citations

TL;DR

Ray unifies training, simulation, and serving for reinforcement learning workloads with a dynamic task/actor model and a scalable, fault-tolerant execution engine; it scales to millions of tasks per second and outperforms specialized systems on RL tasks.

ABSTRACT

The next generation of AI applications will continuously interact with the environment and learn from these interactions. These applications impose new and demanding systems requirements, both in terms of performance and flexibility. In this paper, we consider these requirements and present Ray---a distributed system to address them. Ray implements a unified interface that can express both task-parallel and actor-based computations, supported by a single dynamic execution engine. To meet the performance requirements, Ray employs a distributed scheduler and a distributed and fault-tolerant store to manage the system's control state. In our experiments, we demonstrate scaling beyond 1.8 million tasks per second and better performance than existing specialized systems for several challenging reinforcement learning applications.

Motivation & Objective

• Motivate the need for a unified framework that handles RL workloads (training, simulation, serving) in dynamic environments.
• Propose a single dynamic execution engine that supports both task-parallel and actor-based computations.
• Design a scalable, fault-tolerant system with a distributed scheduler and metadata store to manage control state and lineage.

Proposed method

• Introduce a unified API for tasks (stateless remote functions) and actors (stateful objects).
• Implement a dynamic task graph execution model that automatically triggers computations when inputs are available.
• Architect Ray with a global control store, bottom-up distributed scheduler, and in-memory object store to achieve low latency and scalability.
• Decouple control state from computation via sharded metadata and lineage tracking for fault tolerance.
• Enable nested remote functions and resource-aware scheduling to handle heterogeneous workloads.

Experimental results

Research questions

• RQ1 How can a single framework efficiently support RL workloads that require simulation, distributed training, and policy serving?
• RQ2 What architectural choices enable millisecond-level latency, high throughput, and fault tolerance for dynamic, heterogeneous tasks?
• RQ3 Can a unified actor/task model outperform stitched multi-system approaches for RL applications?

Key findings

• Ray scales beyond 1.8 million tasks per second in experiments.
• Ray achieves millisecond-level latencies with a bottom-up distributed scheduler and sharded metadata store.
• The Global Control Store enables stateless components and scalable fault-tolerant lineage tracking.
• Ray provides locality-aware task placement and near-linear scalability on large clusters.
• Ray demonstrates better performance than existing specialized systems for several RL applications (training, serving, simulation).
• The object store achieves high throughput, with write throughput over 15 GB/s and enables zero-copy data sharing on the same node.