[Paper Review] PySCIPOpt-ML: Embedding Trained Machine Learning Models into Mixed-Integer Programs

Mark Turner, Antonia Chmiela+2 more|arXiv (Cornell University)|Dec 13, 2023

Formal Methods in Verification4 citations

TL;DR

PySCIPOpt-ML is an open-source Python package that automatically formulates trained machine learning models—such as neural networks, tree ensembles, and linear models—into mixed-integer programs (MIPs) for integration into optimization workflows. It enables seamless embedding of ML predictors into MIPs via SCIP, with a focus on numerical stability and broad framework support, and introduces SurrogateLIB, a library of semi-realistic MIP instances with embedded ML constraints.

ABSTRACT

A standard tool for modelling real-world optimisation problems is mixed-integer programming (MIP). However, for many of these problems, information about the relationships between variables is either incomplete or highly complex, making it difficult or even impossible to model the problem directly. To overcome these hurdles, machine learning (ML) predictors are often used to represent these relationships and are then embedded in the MIP as surrogate models. Due to the large amount of available ML frameworks and the complexity of many ML predictors, formulating such predictors into MIPs is a highly non-trivial task. In this paper, we introduce PySCIPOpt-ML, an open-source tool for the automatic formulation and embedding of trained ML predictors into MIPs. By directly interfacing with a broad range of commonly used ML frameworks and an open-source MIP solver, PySCIPOpt-ML provides a way to easily integrate ML constraints into optimisation problems. Alongside PySCIPOpt-ML, we introduce, SurrogateLIB, a library of MIP instances with embedded ML constraints, and present computational results over SurrogateLIB, providing intuition on the scale of ML predictors that can be practically embedded. The project is available at https://github.com/Opt-Mucca/PySCIPOpt-ML.

Motivation & Objective

• To address the challenge of integrating complex or unknown relationships in real-world optimization problems where traditional MIP formulations are infeasible due to incomplete or nonlinear variable dependencies.
• To provide a unified, automated tool for translating trained ML models from popular frameworks into equivalent MIP constraints that preserve numerical stability and solution robustness.
• To support a wide range of ML models—including neural networks, gradient-boosted trees, and linear models—for use within MIPs via direct integration with the SCIP solver.
• To create a reproducible, extensible library of MIP instances with embedded ML constraints to support benchmarking and research in machine learning-enhanced optimization.

Proposed method

• The package directly interfaces with Scikit-Learn, XGBoost, LightGBM, and PyTorch to extract trained models and automatically generate equivalent MIP formulations using piecewise-linear approximations and big-M constraints.
• For neural networks, it employs a mixed-integer programming formulation based on ReLU activation functions, using disjunctive constraints to model the nonlinearity while ensuring numerical stability.
• For tree-based models like random forests and gradient-boosted trees, it uses a branch-and-bound-style formulation that represents each decision path as a set of binary variables and logical constraints.
• The formulation supports both regression and classification tasks, with output variables linked to MIP decision variables through tight, solver-friendly constraints.
• It integrates with the open-source SCIP MIP solver, enabling efficient solution of MIPs that include ML surrogate constraints.
• The SurrogateLIB library is generated by parameterizing a base optimization problem (e.g., auto design) with real-world data, allowing for homogeneous yet diverse instances with varying complexity.

Figure 1: How trained ML predictors can be automatically formulated as MIPs and embedded in an optimisation problem using our Python package PySCIPOpt-ML.

Experimental results

Research questions

• RQ1 How can trained machine learning models from diverse frameworks be automatically and robustly embedded into mixed-integer programs?
• RQ2 What MIP formulation strategies ensure numerical stability and efficient solution when integrating ML predictors into optimization models?
• RQ3 Can a unified software tool support a broad range of ML models—including deep neural networks, tree ensembles, and linear models—within a single MIP framework?
• RQ4 How can realistic, semi-synthetic MIP instances with embedded ML constraints be systematically generated for benchmarking and research?
• RQ5 To what extent does the integration of ML surrogates improve the modeling capability of MIPs in real-world applications with complex or unknown relationships?

Key findings

• PySCIPOpt-ML successfully supports the automatic MIP formulation of diverse ML models, including neural networks with ReLU, sigmoid, and tanh activations, as well as tree-based models from XGBoost, LightGBM, and Scikit-Learn.
• The tool enables end-to-end integration of ML predictors into MIPs via SCIP, with formulations designed to enhance numerical stability and avoid solver convergence issues.
• The SurrogateLIB library provides a collection of 100+ semi-realistic MIP instances with embedded ML constraints, generated from a real-world auto design problem using historical car sales data.
• The formulation of ReLU-based neural networks is achieved using a mixed-integer representation that models each neuron’s piecewise-linear behavior via disjunctive constraints.
• The package supports both regression and classification tasks, with output variables linked to MIP decision variables through tight, constraint-based mappings.
• The tool is open-source and available at https://github.com/Opt-Mucca/PySCIPOpt-ML, with SurrogateLIB accessible at https://zenodo.org/records/10357875.