[Paper Review] Language Models for Business Optimisation with a Real World Case Study in Production Scheduling

Pivithuru Thejan Amarasinghe, Su Nguyen+2 more|arXiv (Cornell University)|Sep 22, 2023

Business Process Modeling and AnalysisBusiness, Management and Accounting3 citations

TL;DR

This paper proposes an AI-Copilot framework that leverages fine-tuned code-generating large language models (LLMs) to automate business optimization problem formulation, particularly for production scheduling. By using prompt engineering and modularization to overcome token limitations, the framework generates accurate, executable optimization models from natural language descriptions, achieving high-quality formulations with minimal expert input.

ABSTRACT

Business optimisation has been used extensively to determine optimal solutions for challenging business operations. Problem formulation is an important part of business optimisation as it influences both the validity of solutions and the efficiency of the optimisation process. While different optimisation modelling languages have been developed, problem formulation is still not a trivial task and usually requires optimisation expertise and problem-domain knowledge. Recently, Large Language Models (LLMs) have demonstrated outstanding performance across different language-related tasks. Since problem formulation can be viewed as a translation task, there is a potential to leverage LLMs to automate problem formulation. However, developing an LLM for problem formulation is challenging, due to limited training data, and the complexity of real-world optimisation problems. Several prompt engineering methods have been proposed in the literature to automate problem formulation with LLMs. While the initial results are encouraging, the accuracy of formulations generated by these methods can still be significantly improved. In this paper, we present an LLM-based framework for automating problem formulation in business optimization. Our approach introduces a method for fine-tuning cost-efficient LLMs specifically tailored to specialized business optimization challenges. The experiment results demonstrate that our framework can generate accurate formulations for conventional and real-world business optimisation problems in production scheduling. Extensive analyses show the effectiveness and the convergence of the proposed fine-tuning method. The proposed method also shows very competitive performance when compared with the state-of-the-art prompt engineering methods in the literature when tested on general linear programming problems.

Motivation & Objective

• To reduce reliance on expert knowledge in formulating complex business optimization problems.
• To address the challenge of token limitations in LLMs when generating large optimization models.
• To develop performance evaluation metrics tailored for assessing the quality of LLM-generated problem formulations.
• To demonstrate the feasibility of using LLMs for end-to-end problem formulation in real-world business scenarios like production scheduling.
• To enable non-expert users to generate correct and executable optimization models without deep optimization expertise.

Proposed method

• Fine-tuning a pre-trained code-generating LLM on a small, curated dataset of problem descriptions and corresponding executable formulations.
• Applying modularization and prompt engineering to break down complex formulations into token-compliant components.
• Designing custom evaluation metrics to assess correctness, completeness, and structural quality of generated formulations.
• Using PCA to visualize and analyze the semantic clustering of encoder and decoder embeddings from the fine-tuned LLM.
• Generating and executing problem formulations using constraint programming solvers to validate correctness and output quality.
• Employing a two-stage process: (1) LLM generates modular code based on natural language input, (2) modules are assembled into a complete, executable model.

Figure 1: Solution overview

Experimental results

Research questions

• RQ1 Can a fine-tuned LLM generate accurate and executable optimization formulations for production scheduling from natural language descriptions?
• RQ2 How can token limitations in LLMs be effectively mitigated when generating large-scale optimization models?
• RQ3 What evaluation metrics are most suitable for assessing the quality of LLM-generated optimization problem formulations?
• RQ4 To what extent do the learned embeddings from the LLM reflect meaningful semantic structures in problem descriptions and formulations?
• RQ5 Can the proposed framework generalize to other types of business optimization problems beyond production scheduling?

Key findings

• The AI-Copilot successfully generated complex, large-scale production scheduling models from natural language inputs, including makespan minimization with random job routes and durations.
• The modularization and prompt engineering approach effectively overcame token limitations, enabling generation of multi-component formulations within LLM context window constraints.
• Embedding analysis revealed that decoder embeddings formed meaningful clusters corresponding to functional modules (e.g., constraints, solution printing), indicating semantic understanding of code structure.
• The framework achieved high accuracy in generating correct problem formulations, as validated by successful execution and output generation on a test case.
• The evaluation metrics designed for this task effectively captured formulation quality, including correctness, completeness, and structural coherence.
• The model demonstrated generalization potential, with the framework adaptable to other optimization problem types such as routing and assignment.

(a) CodeRL does not create a complete problem formulation due to token limitation