[Paper Review] Learning to Dispatch for Job Shop Scheduling via Deep Reinforcement Learning

Cong Zhang, Wen Song+4 more|arXiv (Cornell University)|Oct 23, 2020

Scheduling and Optimization Algorithms49 references236 citations

TL;DR

The paper presents a size-agnostic, graph neural network–based DRL method to automatically learn high-quality priority dispatching rules for solving Job Shop Scheduling Problems, outperforming traditional PDR baselines and generalizing to larger instances.

ABSTRACT

Priority dispatching rule (PDR) is widely used for solving real-world Job-shop scheduling problem (JSSP). However, the design of effective PDRs is a tedious task, requiring a myriad of specialized knowledge and often delivering limited performance. In this paper, we propose to automatically learn PDRs via an end-to-end deep reinforcement learning agent. We exploit the disjunctive graph representation of JSSP, and propose a Graph Neural Network based scheme to embed the states encountered during solving. The resulting policy network is size-agnostic, effectively enabling generalization on large-scale instances. Experiments show that the agent can learn high-quality PDRs from scratch with elementary raw features, and demonstrates strong performance against the best existing PDRs. The learned policies also perform well on much larger instances that are unseen in training.

Motivation & Objective

• Automate the design of effective priority dispatching rules (PDRs) for JSSP using deep reinforcement learning.
• Leverage disjunctive graph representations to capture dependencies and machine status for scheduling decisions.
• Develop a size-agnostic, generalizable policy that scales to varying numbers of jobs and machines.
• Demonstrate that learned PDRs outperform manually designed rules on generated and public benchmarks.

Proposed method

• Formulate JSSP dispatching as a Markov Decision Process where states are disjunctive graphs and actions dispatch eligible operations.
• Embed graph-structured states with a Graph Isomorphism Network (GIN) to obtain fixed-dimensional node and graph embeddings.
• Use an adding-arc strategy to generate sparse directed disjunctive graphs for GNN processing and derive action scores via an MLP.
• Parameterize the policy with a GNN-based network and train using Proximal Policy Optimization (PPO) with a critic sharing the GNN backbone.
• Define the reward as the improvement in the lower bound of makespan, encouraging actions that reduce the projected makespan; use gamma=1 for cumulative reward.
• Evaluate on Taillard-like generated instances and public benchmarks (Taillard and DMU) and compare against traditional PDRs like SPT, MWKR, MOPNR, FDD/MWKR.

Experimental results

Research questions

• RQ1 Can a DRL framework learn high-quality, generalizable PDRs for JSSP from scratch using raw features?
• RQ2 Does a size-agnostic GNN-based policy trained on small instances generalize to larger, unseen problem sizes?
• RQ3 How does the learned PDR perform relative to traditional hand-crafted PDRs on standard benchmarks?
• RQ4 Can the disjunctive graph representation and graph-based embedding effectively capture scheduling dynamics to guide decisions?

Key findings

• The learned PDRs consistently outperform traditional baselines (SPT, MWKR, FDD/MWKR, MOPNR) on generated instances across sizes up to 30×20.
• The learned policy generalizes to substantially larger instances (e.g., 50×20 and 100×20) not seen during training, with performance superior to traditional PDRs.
• Training with PPO and a GIN-based policy yields size-agnostic policies that can scale to arbitrary instance sizes without re-training for each size.
• Inference times are longer than classic PDRs but provide substantial makespan improvements and outperform OR-Tools on many large instances within practical time limits.
• Experiments on Taillard and DMU benchmarks show the learned PDRs maintain strong performance and better gaps to best-known solutions than baselines.