[Paper Review] A Reinforcement Learning Environment For Job-Shop Scheduling

Pierre Tassel, Martin Gebser+1 more|arXiv (Cornell University)|Apr 8, 2021

Scheduling and Optimization Algorithms20 references45 citations

TL;DR

The paper presents a single-agent PPO-based DRL environment for Job-Shop Scheduling with a compact state representation and a dense reward function, achieving better results than dispatching rules and prior RL methods on standard benchmarks, approaching state-of-the-art COP performance.

ABSTRACT

Scheduling is a fundamental task occurring in various automated systems applications, e.g., optimal schedules for machines on a job shop allow for a reduction of production costs and waste. Nevertheless, finding such schedules is often intractable and cannot be achieved by Combinatorial Optimization Problem (COP) methods within a given time limit. Recent advances of Deep Reinforcement Learning (DRL) in learning complex behavior enable new COP application possibilities. This paper presents an efficient DRL environment for Job-Shop Scheduling -- an important problem in the field. Furthermore, we design a meaningful and compact state representation as well as a novel, simple dense reward function, closely related to the sparse make-span minimization criteria used by COP methods. We demonstrate that our approach significantly outperforms existing DRL methods on classic benchmark instances, coming close to state-of-the-art COP approaches.

Motivation & Objective

• Motivate applying deep reinforcement learning to Job-Shop Scheduling (JSS) to handle intractable COP instances and enable lifelong learning across related instances.
• Propose a single-agent dispatcher formulation for JSS and train a policy with Proximal Policy Optimization (PPO).
• Design a compact, informative state representation and a dense reward closely tied to make-span minimization.
• Introduce search-space reduction and problem symmetry considerations to improve learning efficiency.
• Demonstrate empirical performance improvements over dispatching rules and prior RL approaches on benchmark instances.

Proposed method

• Model JSS as a single-agent Markov Decision Process where the agent selects which job to schedule next or chooses No-Op to advance time.
• Use Proximal Policy Optimization (PPO) with separate MLPs for the policy and value function, trained with clipped objective to stabilize updates.
• Provide a dense reward R(s,a) based on scheduled operation length minus machine idle time, scaled by the maximum operation length; relate this density to make-span minimization.
• Introduce a compact state representation as a 7-attribute per-job matrix to capture allocatability, remaining times, and idle metrics, enabling straightforward dispatching-rule interpretations (e.g., MWKR, FIFO).
• Employ action masking to forbid illegal actions, and apply search-space reductions such as non-final prioritization and No-Op restrictions to guide exploration.
• Implement the environment with OpenAI Gym, train with RLLib/TensorFlow, and perform hyperparameter tuning via WandB.

Experimental results

Research questions

• RQ1 How effective is a single-agent PPO-based DRL approach for solving JSS under time constraints?
• RQ2 Can a compact, interpretable state representation and a dense reward accelerate learning and performance relative to prior RL and dispatching-rule methods?
• RQ3 What is the impact of search-space reductions and No-Op handling on learning efficiency and solution quality?
• RQ4 How well does the trained agent generalize across benchmark datasets (Taillard and Demirkol) with fixed hyperparameters?

Key findings

• The PPO-based DRL approach yields solutions superior to FIFO and MWKR dispatching rules on all Taillard and Demirkol instances evaluated.
• Average make-span improvement over MWKR on Taillard is 11% and 12% on Demirkol instances.
• The method approaches, but does not surpass, OR-Tools CP solver performance, indicating strong competitive results for DRL in this setting.
• The environment and training setup produce results that exceed previous RL methods reported in the literature for JSS on the same benchmarks.
• The learned policy demonstrates generalization across datasets with similar problem structures despite instance-specific training.
• The dense reward correlates with make-span improvements, supporting its suitability for guiding DRL in scheduling tasks.