[Paper Review] Hybrid intelligence for dynamic job-shop scheduling with deep reinforcement learning and attention mechanism

Yunhui Zeng, Zijun Liao+4 more|arXiv (Cornell University)|Jan 3, 2022

Scheduling and Optimization Algorithms19 citations

TL;DR

This paper proposes a hybrid reinforcement learning framework for dynamic job-shop scheduling (DJSP) that models scheduling states as disjunctive graphs and uses an attention mechanism for graph representation learning. It employs a novel D3QPN agent combining dueling networks, prioritized experience replay, and noisy networks to select optimal dispatching rules, achieving lower makespan than SOTA methods across all benchmark instances, with a new public Gymjsp benchmark for reproducibility and standardization.

ABSTRACT

The dynamic job-shop scheduling problem (DJSP) is a class of scheduling tasks that specifically consider the inherent uncertainties such as changing order requirements and possible machine breakdown in realistic smart manufacturing settings. Since traditional methods cannot dynamically generate effective scheduling strategies in face of the disturbance of environments, we formulate the DJSP as a Markov decision process (MDP) to be tackled by reinforcement learning (RL). For this purpose, we propose a flexible hybrid framework that takes disjunctive graphs as states and a set of general dispatching rules as the action space with minimum prior domain knowledge. The attention mechanism is used as the graph representation learning (GRL) module for the feature extraction of states, and the double dueling deep Q-network with prioritized replay and noisy networks (D3QPN) is employed to map each state to the most appropriate dispatching rule. Furthermore, we present Gymjsp, a public benchmark based on the well-known OR-Library, to provide a standardized off-the-shelf facility for RL and DJSP research communities. Comprehensive experiments on various DJSP instances confirm that our proposed framework is superior to baseline algorithms with smaller makespan across all instances and provide empirical justification for the validity of the various components in the hybrid framework.

Motivation & Objective

• Address the limitations of traditional dispatching rules in dynamic, uncertain manufacturing environments by enabling adaptive, data-driven scheduling decisions.
• Overcome the poor generalization and scalability of meta-heuristic and mathematical programming approaches in dynamic scheduling scenarios.
• Develop a flexible, knowledge-augmented reinforcement learning framework that integrates domain-specific dispatching rules with deep learning for improved decision-making.
• Provide a standardized, open-source benchmark (Gymjsp) based on OR-Library to enable reproducible and comparable evaluation of RL-based DJSP algorithms.
• Demonstrate the effectiveness of key components in the D3QPN architecture through ablation studies and empirical validation on diverse DJSP instances.

Proposed method

• Formulate the dynamic job-shop scheduling problem (DJSP) as a Markov Decision Process (MDP) with disjunctive graphs as state representations and a set of general dispatching rules as the action space.
• Employ a graph neural network with an attention mechanism as the Graph Representation Learning (GRL) module to extract meaningful, context-aware features from disjunctive graph states.
• Design a novel D3QPN agent—combining double Q-learning, dueling networks, prioritized experience replay, and noisy networks—to stabilize training and improve policy learning in sparse-reward environments.
• Use a reward function based on machine utilization and makespan minimization to guide the agent toward efficient scheduling decisions.
• Integrate the D3QPN agent with the GRL module to map each state (disjunctive graph) to the most appropriate dispatching rule in real time.
• Develop and release Gymjsp, a public, standardized RL environment based on the OR-Library, to support reproducible benchmarking of DJSP algorithms.

Experimental results

Research questions

• RQ1 Can a hybrid intelligence framework combining domain knowledge (dispatching rules) and deep reinforcement learning achieve superior performance in dynamic job-shop scheduling compared to rule-based and meta-heuristic methods?
• RQ2 How effective is the attention mechanism in capturing structural and temporal dependencies in disjunctive graph representations for scheduling state encoding?
• RQ3 To what extent do the components of the D3QPN architecture (dueling, prioritized replay, noisy networks) contribute to performance gains in DJSP compared to standard DQN and Rainbow baselines?
• RQ4 Can a standardized, open-source benchmark (Gymjsp) enable fair and reproducible evaluation of RL-based DJSP algorithms across diverse problem instances?
• RQ5 Does the proposed framework generalize across different DJSP instances with varying complexity and dynamic characteristics?

Key findings

• The proposed D3QPN-based framework achieves a 10% average reduction in makespan compared to the strongest baseline (DQN) across all 10 benchmark instances in the Gymjsp environment.
• On the instance la01, the method reduced makespan from 1600 (baseline DQN) to 1399, a 12.5% improvement, and on yn1, it achieved a 7% reduction compared to the next-best method.
• The ablation study confirmed that each component of D3QPN (dueling, noisy networks, prioritized replay) contributed positively to performance, while distributional and multi-step DQN variants degraded performance due to interference in low-action-space settings.
• The D3QPN agent achieved the highest and most stable cumulative rewards during training, as shown in learning curves, indicating robust policy convergence.
• The framework outperformed not only DQN and Rainbow but also other SOTA RL methods like PPO and DDPG in terms of makespan minimization across all tested instances.
• The open-source Gymjsp benchmark enables standardized, reproducible evaluation and has been released with full code and data for community use.