[Paper Review] How to effectively use machine learning models to predict the solutions for optimization problems: lessons from loss function

Mahdi Abolghasemi, Babak Abbasi+2 more|arXiv (Cornell University)|May 14, 2021

Constraint Satisfaction and Optimization33 references6 citations

TL;DR

This paper proposes using advanced machine learning models—particularly LightGBM with mean absolute error (MAE) loss—to predict optimal solutions for large-scale stochastic optimization problems, such as blood transshipment. It demonstrates that MAE loss significantly improves constraint adherence and solution accuracy (up to 98% similarity to optimal solutions), outperforming MSE and Huber loss, while enabling faster, cost-effective predictions than commercial solvers.

ABSTRACT

Using machine learning in solving constraint optimization and combinatorial problems is becoming an active research area in both computer science and operations research communities. This paper aims to predict a good solution for constraint optimization problems using advanced machine learning techniques. It extends the work of \cite{abbasi2020predicting} to use machine learning models for predicting the solution of large-scaled stochastic optimization models by examining more advanced algorithms and various costs associated with the predicted values of decision variables. It also investigates the importance of loss function and error criterion in machine learning models where they are used for predicting solutions of optimization problems. We use a blood transshipment problem as the case study. The results for the case study show that LightGBM provides promising solutions and outperforms other machine learning models used by \cite{abbasi2020predicting} specially when mean absolute deviation criterion is used.

Motivation & Objective

• To develop a machine learning framework that predicts optimal decision variables for large-scale stochastic optimization problems, particularly in supply chain contexts.
• To investigate the impact of different loss functions—MAE, MSE, and Huber—on the accuracy and constraint adherence of ML-predicted solutions.
• To evaluate the practical utility of ML-predicted solutions by analyzing key supply chain costs (ordering, holding, transshipment, shortage, outdate).
• To compare the performance of multiple ML models (LightGBM, SVR, Ridge, MLP) in predicting solutions for a real-world blood transshipment problem.
• To demonstrate that ML models can generate near-optimal solutions faster and more cheaply than commercial optimization solvers.

Proposed method

• Trained multi-output machine learning models (LightGBM, SVR, Ridge, MLP) on historical data from a two-stage stochastic optimization problem in blood supply chain management.
• Employed three distinct loss functions—Mean Absolute Error (MAE), Mean Squared Error (MSE), and Huber loss—for training LightGBM to assess their impact on solution quality and constraint compliance.
• Evaluated model performance using both solution accuracy (similarity to optimal solutions) and constraint violation rates across 814,000 order and inventory records.
• Computed and compared total supply chain costs (ordering, holding, transshipment, shortage, outdate) under predictions from each model to assess forecast utility.
• Used a case study of a blood transshipment network with 4 hospitals and 12 blood product ages to simulate real-world stochastic demand and perishability.
• Optimized models using hyperparameter tuning and validated predictions against exact solutions from an optimization solver.

Experimental results

Research questions

• RQ1 How does the choice of loss function (MAE, MSE, Huber) affect the accuracy and constraint adherence of ML models predicting solutions to constrained optimization problems?
• RQ2 Which machine learning model (LightGBM, SVR, Ridge, MLP) performs best in predicting optimal decision variables for a large-scale stochastic optimization problem?
• RQ3 To what extent do ML-predicted solutions match the optimal solutions in terms of solution similarity and cost efficiency?
• RQ4 Can ML models generate feasible solutions that violate constraints less frequently than others, especially when trained with different loss functions?
• RQ5 How do the total supply chain costs (ordering, holding, transshipment, shortage, outdate) compare between ML-predicted solutions and optimal solutions?

Key findings

• LightGBM trained with MAE loss achieved up to 98% similarity to the optimal solution, outperforming other models and loss functions in accuracy.
• The LightGBM-MAE model violated constraints in only 21 out of 814,000 records, the lowest among all models, indicating superior constraint adherence.
• Models trained with MSE loss violated constraints significantly more often (e.g., 71 violations for LightGBM-MSE), making it less suitable for constrained optimization prediction.
• SVR and LightGBM-MAE both achieved only 21 constraint violations, demonstrating strong feasibility in predicted solutions.
• ML models, particularly LightGBM-MAE, generated competitive total costs while reducing specific costs like shortage and outdate, showing utility beyond just cost parity.
• The MLP model performed poorly, violating constraints 411 times and showing the lowest accuracy, indicating limited suitability for this task.