[论文解读] Language Models for Business Optimisation with a Real World Case Study in Production Scheduling

Pivithuru Thejan Amarasinghe, Su Nguyen+2 more|arXiv (Cornell University)|Sep 22, 2023

Business Process Modeling and AnalysisBusiness, Management and Accounting被引用 3

一句话总结

本文提出了一种AI-Copilot框架，利用微调后的代码生成大型语言模型（LLMs）自动化业务优化问题的建模，尤其针对生产排程问题。通过提示工程和模块化设计克服上下文长度限制，该框架能够从自然语言描述中生成准确且可执行的优化模型，仅需极少专家输入即可实现高质量建模。

ABSTRACT

Business optimisation has been used extensively to determine optimal solutions for challenging business operations. Problem formulation is an important part of business optimisation as it influences both the validity of solutions and the efficiency of the optimisation process. While different optimisation modelling languages have been developed, problem formulation is still not a trivial task and usually requires optimisation expertise and problem-domain knowledge. Recently, Large Language Models (LLMs) have demonstrated outstanding performance across different language-related tasks. Since problem formulation can be viewed as a translation task, there is a potential to leverage LLMs to automate problem formulation. However, developing an LLM for problem formulation is challenging, due to limited training data, and the complexity of real-world optimisation problems. Several prompt engineering methods have been proposed in the literature to automate problem formulation with LLMs. While the initial results are encouraging, the accuracy of formulations generated by these methods can still be significantly improved. In this paper, we present an LLM-based framework for automating problem formulation in business optimization. Our approach introduces a method for fine-tuning cost-efficient LLMs specifically tailored to specialized business optimization challenges. The experiment results demonstrate that our framework can generate accurate formulations for conventional and real-world business optimisation problems in production scheduling. Extensive analyses show the effectiveness and the convergence of the proposed fine-tuning method. The proposed method also shows very competitive performance when compared with the state-of-the-art prompt engineering methods in the literature when tested on general linear programming problems.

研究动机与目标

• 减少在复杂业务优化问题建模过程中对专家知识的依赖。
• 解决LLM在生成大规模优化模型时面临的上下文长度限制问题。
• 设计适用于评估LLM生成问题建模质量的性能评价指标。
• 证明LLM在真实业务场景（如生产排程）中实现端到端问题建模的可行性。
• 使非专家用户无需具备深入的优化专业知识，也能生成正确且可执行的优化模型。

提出的方法

• 在小规模、精心筛选的问题描述与对应可执行建模代码数据集上，对预训练的代码生成LLM进行微调。
• 采用模块化设计与提示工程，将复杂建模任务分解为符合上下文长度限制的组件。
• 设计自定义评价指标，用于评估生成建模的正确性、完整性与结构质量。
• 使用主成分分析（PCA）可视化并分析微调后LLM的编码器与解码器嵌入的语义聚类。
• 使用约束规划求解器生成并执行问题建模，以验证其正确性与输出质量。
• 采用两阶段流程：(1) LLM根据自然语言输入生成模块化代码；(2) 将模块组合为完整且可执行的模型。

Figure 1: Solution overview

实验结果

研究问题

• RQ1 微调后的LLM能否从自然语言描述中生成准确且可执行的生产排程优化建模？
• RQ2 在生成大规模优化模型时，如何有效缓解LLM的上下文长度限制？
• RQ3 哪些评价指标最适用于评估LLM生成的优化问题建模质量？
• RQ4 LLM学习到的嵌入在多大程度上反映了问题描述与建模之间的有意义语义结构？
• RQ5 所提出的框架能否推广到生产排程以外的其他类型业务优化问题？

主要发现

• AI-Copilot成功从自然语言输入中生成了复杂且大规模的生产排程模型，包括具有随机作业路径与持续时间的完工时间最小化问题。
• 模块化设计与提示工程方法有效克服了上下文长度限制，使在LLM上下文窗口约束内生成多组件建模成为可能。
• 嵌入分析显示，解码器嵌入形成了与功能模块（如约束、结果输出）相对应的有意义聚类，表明模型对代码结构具有语义理解能力。
• 该框架在生成正确问题建模方面表现出高准确率，经测试案例验证，能成功执行并生成正确输出。
• 为本任务设计的评价指标有效捕捉了建模质量，包括正确性、完整性与结构一致性。
• 模型展现出良好的泛化潜力，该框架可适配其他优化问题类型，如路径规划与指派问题。

(a) CodeRL does not create a complete problem formulation due to token limitation