[Paper Review] RLScheduler: An Automated HPC Batch Job Scheduler Using Reinforcement Learning

Di Zhang, Dong Dai+3 more|arXiv (Cornell University)|Oct 20, 2019

Cloud Computing and Resource Management42 references4 citations

TL;DR

RLScheduler is a reinforcement learning-based HPC batch job scheduler that automatically learns high-quality scheduling policies through trial-and-error, requiring minimal expert intervention. It outperforms traditional schedulers across diverse workloads and optimization goals, generalizing effectively to unseen workloads with stable performance using a kernel-based neural network and trajectory filtering for efficient, robust learning.

ABSTRACT

Today high-performance computing (HPC) platforms are still dominated by batch jobs. Accordingly, effective batch job scheduling is crucial to obtain high system efficiency. Existing HPC batch job schedulers typically leverage heuristic priority functions to prioritize and schedule jobs. But, once configured and deployed by the experts, such priority functions can hardly adapt to the changes of job loads, optimization goals, or system settings, potentially leading to degraded system efficiency when changes occur. To address this fundamental issue, we present RLScheduler, an automated HPC batch job scheduler built on reinforcement learning. RLScheduler relies on minimal manual interventions or expert knowledge, but can learn high-quality scheduling policies via its own continuous 'trial and error'. We introduce a new kernel-based neural network structure and trajectory filtering mechanism in RLScheduler to improve and stabilize the learning process. Through extensive evaluations, we confirm that RLScheduler can learn high-quality scheduling policies towards various workloads and various optimization goals with relatively low computation cost. Moreover, we show that the learned models perform stably even when applied to unseen workloads, making them practical for production use.

Motivation & Objective

• To address the inflexibility of traditional HPC schedulers that rely on fixed, manually tuned priority functions.
• To enable automated, adaptive job scheduling that dynamically responds to changing workloads and optimization goals.
• To evaluate whether reinforcement learning can learn stable, high-performance scheduling policies without expert intervention.
• To identify key factors affecting RL-based scheduler performance and propose solutions for improved learning efficiency and robustness.

Proposed method

• RLScheduler employs a deep Q-network (DQN) agent trained via reinforcement learning to assign job priorities based on job attributes such as submission time, requested processors, and runtime.
• It introduces a kernel-based deep neural network to improve policy representation and generalization across diverse job workloads.
• A trajectory filtering mechanism is used to reduce noise and variance in the training data, stabilizing the learning process.
• The scheduler is trained using sparse rewards based on system-level metrics like job throughput, wait time, and resource utilization.
• Training is conducted on synthetic and real-world job traces, with evaluation on unseen workloads to test generalization.
• The system is evaluated under both backfilling and non-backfilling conditions to assess robustness across deployment scenarios.

Experimental results

Research questions

• RQ1 Can reinforcement learning learn high-quality scheduling policies that match or exceed state-of-the-art heuristic schedulers across various workloads and optimization goals?
• RQ2 Does the learned RL policy generalize effectively to unseen job workloads not seen during training?
• RQ3 What are the key factors affecting the learning efficiency and performance stability of RL-based HPC schedulers?
• RQ4 How do neural network architecture and training data variance influence the scalability and robustness of the learned scheduling policy?

Key findings

• RLScheduler achieves scheduling performance comparable to or better than state-of-the-art schedulers like FCFS, SJF, WFP3, and UNICEP across multiple real and synthetic workloads.
• On the Lublin-1 trace with backfilling, RLScheduler reduced average job waiting time to 12,460 seconds, outperforming FCFS (24,887 s) and other schedulers.
• For job slowdown, RLScheduler achieved 270.7 on HPC2N (without backfilling), significantly outperforming FCFS (13,938) and UNICEP (29,633).
• The model generalized well to unseen workloads, maintaining stable performance even when trained on one trace and tested on another, demonstrating strong generalization capability.
• The kernel-based neural network and trajectory filtering mechanism significantly improved training stability and convergence speed, reducing variance in policy learning.
• RLScheduler achieved low computation cost during training and maintained high performance across diverse metrics, including throughput, wait time, and slowdown.