[Paper Review] Generalizable Resource Allocation in Stream Processing via Deep Reinforcement Learning

Xiang Ni, Jing Li+3 more|arXiv (Cornell University)|Nov 19, 2019

Caching and Content Delivery26 references4 citations

TL;DR

This paper proposes a graph-aware deep reinforcement learning (DRL) framework for generalizable resource allocation in stream processing systems. By leveraging graph embedding to capture structural properties and a context-aware decoder to generate device assignments, the model outperforms METIS and LSTM-based baselines on 70% of unseen stream graphs, demonstrating strong generalization to novel topologies.

ABSTRACT

This paper considers the problem of resource allocation in stream processing, where continuous data flows must be processed in real time in a large distributed system. To maximize system throughput, the resource allocation strategy that partitions the computation tasks of a stream processing graph onto computing devices must simultaneously balance workload distribution and minimize communication. Since this problem of graph partitioning is known to be NP-complete yet crucial to practical streaming systems, many heuristic-based algorithms have been developed to find reasonably good solutions. In this paper, we present a graph-aware encoder-decoder framework to learn a generalizable resource allocation strategy that can properly distribute computation tasks of stream processing graphs unobserved from training data. We, for the first time, propose to leverage graph embedding to learn the structural information of the stream processing graphs. Jointly trained with the graph-aware decoder using deep reinforcement learning, our approach can effectively find optimized solutions for unseen graphs. Our experiments show that the proposed model outperforms both METIS, a state-of-the-art graph partitioning algorithm, and an LSTM-based encoder-decoder model, in about 70% of the test cases.

Motivation & Objective

• Address the challenge of generalizable resource allocation in stream processing systems where graph structures vary widely and are unseen during training.
• Overcome the limitations of heuristic-based graph partitioning tools like METIS, which struggle with optimal partition count selection and lack generalization across diverse graph topologies.
• Develop a deep reinforcement learning model that learns a transferable policy for operator-to-device mapping in stream processing DAGs.
• Capture complex dependencies between computation workloads, communication costs, and device load balance through structured state representations.
• Enable end-to-end learning of resource allocation policies that generalize to unseen stream graphs without retraining

Proposed method

• Employ a graph autoencoder with graph attention networks (GAT) to learn node-level and global graph embeddings that encode structural and workload characteristics of stream processing graphs.
• Design a graph-aware decoder that generates device assignments sequentially, attending to both the current partial assignment and the global graph representation to model inter-operator dependencies.
• Formulate the resource allocation problem as a sequential decision-making process in DRL, where the agent selects device assignments for operators one by one to maximize long-term throughput.
• Train the encoder-decoder model jointly using deep reinforcement learning with a dense reward function that balances load distribution and minimizes inter-device communication.
• Use a policy gradient method (e.g., PPO or A3C) to optimize the agent’s strategy based on performance feedback from simulated stream processing workloads.
• Integrate graph-level global features such as critical path length and total communication cost into the state representation to guide the learning of performance-aware policies.

Experimental results

Research questions

• RQ1 Can a deep reinforcement learning agent learn a generalizable policy for resource allocation across diverse, unseen stream processing graph topologies?
• RQ2 How does graph embedding improve the representation of structural and workload features in stream graphs compared to handcrafted or sequential features?
• RQ3 To what extent does a graph-aware decoder outperform standard sequence-to-sequence models in modeling complex dependencies between operator placements?
• RQ4 Does the proposed DRL-based approach generalize better than existing graph partitioning tools like METIS and sequence-based models on unobserved graphs?
• RQ5 How effective is the joint training of graph embedding and DRL policy in balancing load and minimizing communication in real-time stream processing?

Key findings

• The proposed DRL framework outperforms the METIS graph partitioning library in 70% of test cases on unseen stream graphs, demonstrating superior generalization.
• The model achieves better system throughput than an LSTM-based encoder-decoder baseline on 70% of unobserved graphs, highlighting the advantage of graph-aware modeling.
• Graph embedding effectively captures structural and workload-related features of stream processing graphs, enabling better state representation than sequential or handcrafted features.
• The graph-aware decoder significantly improves the modeling of inter-operator dependencies and device assignment constraints compared to standard sequence models.
• Joint training of the graph encoder and DRL policy enables the model to learn a transferable strategy that generalizes across diverse graph topologies without retraining.
• The framework successfully balances workload distribution and minimizes inter-device communication, leading to higher end-to-end throughput in real-time stream processing workloads.