[Paper Review] RLlib Flow: Distributed Reinforcement Learning is a Dataflow Problem

Eric Liang, Zhanghao Wu+4 more|arXiv (Cornell University)|Nov 25, 2020

Reinforcement Learning in Robotics29 references4 citations

TL;DR

This paper proposes RLlib Flow, a hybrid actor-dataflow programming model that reframes distributed reinforcement learning as a dataflow problem, enabling high-level, composable, and performant implementations. By combining dataflow abstractions with controlled actor message passing, it reduces production code by 2–9× and supports complex multi-agent algorithms previously unattainable without low-level systems coding.

ABSTRACT

Researchers and practitioners in the field of reinforcement learning (RL) frequently leverage parallel computation, which has led to a plethora of new algorithms and systems in the last few years. In this paper, we re-examine the challenges posed by distributed RL and try to view it through the lens of an old idea: distributed dataflow. We show that viewing RL as a dataflow problem leads to highly composable and performant implementations. We propose RLlib Flow, a hybrid actor-dataflow programming model for distributed RL, and validate its practicality by porting the full suite of algorithms in RLlib, a widely adopted distributed RL library. Concretely, RLlib Flow provides 2-9 code savings in real production code and enables the composition of multi-agent algorithms not possible by end users before. The open-source code is available as part of RLlib at https://github.com/ray-project/ray/tree/master/rllib.

Motivation & Objective

• To address the high barrier to entry and limited extensibility in existing distributed RL libraries, which often require low-level message passing and lack composability.
• To enable researchers and practitioners to easily customize, debug, and compose distributed RL algorithms without deep systems expertise.
• To reduce the complexity and boilerplate in production RL code while maintaining high performance.
• To support novel use cases such as interleaved training of different algorithms and complex multi-agent training patterns.
• To demonstrate that a high-level dataflow abstraction can match or exceed the performance of low-level actor-based implementations.

Proposed method

• The authors design a hybrid actor-dataflow model that composes dataflow operators with explicit references to actor processes, enabling controlled message passing.
• They introduce sequencing and concurrency operators (e.g., Union, barrier semantics) to manage synchronization between dataflow fragments and actors.
• The model is implemented on Ray, leveraging its actor and distributed task execution primitives to expose high-level abstractions while preserving performance.
• Key primitives include dataflow operators for parallel data processing and actor references for targeted communication, allowing fine-grained control over distributed execution.
• The system supports both synchronous and asynchronous training patterns by expressing algorithmic topologies as composable dataflow graphs with embedded actor interactions.
• The approach is validated by porting all RLlib algorithms to RLlib Flow, demonstrating code reduction and performance equivalence to original low-level implementations.

Experimental results

Research questions

• RQ1 Can distributed reinforcement learning be effectively modeled as a dataflow problem to improve composability and developer productivity?
• RQ2 To what extent can a hybrid actor-dataflow model reduce the complexity of implementing distributed RL algorithms without sacrificing performance?
• RQ3 Can this model enable new algorithmic compositions—such as multi-agent or meta-learning workflows—that were previously infeasible with existing high-level RL libraries?
• RQ4 How does the performance of the high-level dataflow abstraction compare to low-level actor-based implementations in real-world RL workloads?
• RQ5 Can the model support complex, user-defined distributed patterns (e.g., interleaved training) that are not expressible through predefined templates?

Key findings

• RLlib Flow reduces the lines of code required for distributed execution in RLlib by a factor of 2 to 9×, significantly lowering implementation and maintenance overhead.
• The model enables the composition of multi-agent RL algorithms that were previously infeasible to implement without writing low-level systems code.
• Performance benchmarks show RLlib Flow achieves close to theoretical optimal performance, with throughput comparable to or exceeding that of Spark Streaming in PPO training on CartPole-v0.
• The system maintains performance parity with original RLlib implementations built on low-level actor and RPC primitives, demonstrating that high-level abstractions need not incur performance penalties.
• The hybrid model successfully integrates dataflow abstractions with targeted actor message passing, enabling both composability and efficient synchronization in complex training workflows.
• The open-source implementation of RLlib Flow is available as part of the RLlib library, enabling adoption and extension by the broader research and practitioner community.