[Paper Review] Interior Point Solving for LP-based prediction+optimisation

Jayanta Mandi, Tias Guns|arXiv (Cornell University)|Oct 26, 2020

Advanced Optimization Algorithms ResearchMathematics27 references34 citations

TL;DR

The paper introduces IntOpt, an interior-point based differentiable LP layer for end-to-end predict-and-optimize, differentiating through the homogeneous self-dual LP formulation with a log-barrier, and demonstrates competitive results against state-of-the-art methods on MILP-related tasks.

ABSTRACT

Solving optimization problems is the key to decision making in many real-life analytics applications. However, the coefficients of the optimization problems are often uncertain and dependent on external factors, such as future demand or energy or stock prices. Machine learning (ML) models, especially neural networks, are increasingly being used to estimate these coefficients in a data-driven way. Hence, end-to-end predict-and-optimize approaches, which consider how effective the predicted values are to solve the optimization problem, have received increasing attention. In case of integer linear programming problems, a popular approach to overcome their non-differentiabilty is to add a quadratic penalty term to the continuous relaxation, such that results from differentiating over quadratic programs can be used. Instead we investigate the use of the more principled logarithmic barrier term, as widely used in interior point solvers for linear programming. Specifically, instead of differentiating the KKT conditions, we consider the homogeneous self-dual formulation of the LP and we show the relation between the interior point step direction and corresponding gradients needed for learning. Finally our empirical experiments demonstrate our approach performs as good as if not better than the state-of-the-art QPTL (Quadratic Programming task loss) formulation of Wilder et al. and SPO approach of Elmachtoub and Grigas.

Motivation & Objective

• Motivate end-to-end learning for MILP where coefficients are uncertain and predicted from data.
• Propose a differentiable LP relaxation using a log-barrier within an interior-point framework.
• Differentiate through the homogeneous self-dual embedding to obtain task-loss gradients.
• Evaluate IntOpt against state-of-the-art approaches on MILP-related problems and show competitive performance.

Proposed method

• Use LP as the final differentiable layer atop a neural network, enabling end-to-end training for MILP problems.
• Insert a log-barrier term into the LP to make the objective twice differentiable, replacing quadratic penalties.
• Differentiate the homogeneous self-dual (HSD) embedding of the LP rather than the KKT conditions to obtain gradients.
• Employ a forward interior-point pass to compute the LP solution and a backward pass to compute ∂x*/∂c for backpropagation.
• Address numerical stability with early stopping (lambda-cutoff) and damping in linear system solves.
• Use a forward-backward scheme (Eq. 9-12) to compute primal/dual directions and gradients with a damped system when needed.

Experimental results

Research questions

• RQ1 Can an interior-point LP formulation with a log-barrier provide differentiable gradients suitable for end-to-end learning in predict-and-optimize settings?
• RQ2 Does differentiating the homogeneous self-dual LP embedding yield competitive gradients for MILP task losses compared to quadratic/ SPO-based approaches?
• RQ3 How do initialization, stopping criteria (lambda-cutoff), and damping affect numerical stability and performance in end-to-end training?
• RQ4 What is the empirical performance of the proposed IntOpt method relative to two-stage, QPTL, and SPO approaches across various MILP-like tasks?

Key findings

• IntOpt matches or improves upon state-of-the-art methods (QPTL and SPO) while staying more closely aligned with LP solving in the gradient flow.
• Across experiments (knapsack, energy scheduling, shortest path), IntOpt achieves competitive regret with comparable or slightly worse MSE in some cases, illustrating task-focused learning benefits.
• Using the homogeneous self-dual LP formulation and log-barrier yields a differentiable pathway for backpropagation that can be integrated as a neural network layer.
• The HSD-based gradients can be computed efficiently using the same equation-system structure as the forward Newton step, enabling reuse of factorizations and accelerations.
• Hyperparameters such as the lambda-cutoff and damping factor significantly impact numerical stability and performance; stopping early can reduce iterations and improve stability.
• The approach demonstrates that end-to-end learning can leverage LP-like structure to handle uncertainty in coefficients while maintaining strong optimization performance.