[Paper Review] LLMs can Schedule

Henrik Abgaryan, Ararat Harutyunyan+1 more|arXiv (Cornell University)|Aug 13, 2024

Genetics, Bioinformatics, and Biomedical ResearchBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This paper introduces the first supervised 120k dataset of natural language descriptions for job shop scheduling problems (JSSP), enabling fine-tuned LLMs like Phi-3 to achieve scheduling performance comparable to dedicated neural networks. With LoRA fine-tuning and a sampling strategy, the LLM reduces average gap to optimal solution to 8.92%, outperforming baseline methods and demonstrating strong potential for LLMs in combinatorial optimization.

ABSTRACT

The job shop scheduling problem (JSSP) remains a significant hurdle in optimizing production processes. This challenge involves efficiently allocating jobs to a limited number of machines while minimizing factors like total processing time or job delays. While recent advancements in artificial intelligence have yielded promising solutions, such as reinforcement learning and graph neural networks, this paper explores the potential of Large Language Models (LLMs) for JSSP. We introduce the very first supervised 120k dataset specifically designed to train LLMs for JSSP. Surprisingly, our findings demonstrate that LLM-based scheduling can achieve performance comparable to other neural approaches. Furthermore, we propose a sampling method that enhances the effectiveness of LLMs in tackling JSSP.

Motivation & Objective

• To explore the feasibility of using Large Language Models (LLMs) for end-to-end job shop scheduling (JSSP), a classic NP-hard optimization problem.
• To address the lack of specialized datasets for training LLMs on scheduling tasks by creating a novel, natural language-formatted dataset of 120k JSSP instances and solutions.
• To evaluate whether minimal fine-tuning of LLMs can yield competitive scheduling performance compared to dedicated neural network approaches.
• To enhance LLM scheduling performance through a proposed sampling method that selects the best solution from multiple generations.
• To provide a new, interpretable, and generalizable framework for applying LLMs to complex scheduling problems.

Proposed method

• The authors created a supervised dataset of 120k JSSP instances, each represented as natural language descriptions of jobs, machines, processing times, and optimal schedules.
• The dataset is specifically designed for LLM fine-tuning, using natural language rather than matrix-based representations to align with LLMs' training distribution.
• A Phi-3 base LLM was fine-tuned using LoRA (Low-Rank Adaptation) with a small number of trainable parameters to adapt to the JSSP task.
• A sampling strategy was implemented, generating multiple candidate schedules per problem and selecting the one with the lowest makespan (C_max) to improve solution quality.
• The method was evaluated on small-scale JSSP benchmarks, comparing the fine-tuned LLM against existing neural baselines such as L2D and SLJ models.
• Performance was measured using the relative gap to the optimal solution (C_max), with statistical analysis across multiple runs and sample sizes.

Figure 1: Left: Training and Validation Losses of Phi-3 Model. Right: The Norm of the Gradient during the fine-tuning of Phi-3 Model

Experimental results

Research questions

• RQ1 Can Large Language Models (LLMs) be effectively fine-tuned to solve job shop scheduling problems (JSSP) using natural language representations?
• RQ2 How does the performance of a fine-tuned LLM compare to dedicated neural network models (e.g., L2D, SLJ) on small-scale JSSP instances?
• RQ3 To what extent does a sampling strategy improve the solution quality of LLM-generated schedules in JSSP?
• RQ4 Can a general-purpose LLM like Phi-3, with minimal fine-tuning, achieve competitive results in a combinatorial optimization task traditionally dominated by specialized models?
• RQ5 What is the impact of dataset format (natural language vs. matrix) on LLM training and performance for scheduling tasks?

Key findings

• The fine-tuned Phi-3 model achieved an average gap of 8.92% from the optimal solution, significantly outperforming the original L2D method (13.01% gap).
• The sampling strategy with s=10 reduced the median gap and variability, improving consistency and solution quality compared to greedy decoding.
• The SLJ models with β values of 32, 128, and 256 achieved lower median gaps (1.17%, 0.26%, 0.65%) than the Phi-3 model, indicating strong baseline performance.
• Despite being a general-purpose language model, Phi-3 with LoRA fine-tuning and sampling achieved competitive performance, suggesting strong transferability to scheduling tasks.
• The proposed 120k natural language dataset enables effective LLM training for JSSP, demonstrating that LLMs can reason over structured scheduling problems using textual input.
• The results indicate that LLMs, when properly fine-tuned and combined with sampling, are a viable and promising alternative to specialized neural architectures in JSSP.

Figure 2: Box plot comparison of different models: (1) L2D original with greedy inference using a 20x20 policy network (L2D Gap original), (2) L2D policy network with sampling ( s=10 ) (L2D Gap s=10), (3) SLJ models trained with $\beta$ values of 32, 128, and 256, and evaluated with sampling ( s=10