[Paper Review] Quasi-Newton approaches to Interior Point Methods for quadratic problems

Jacek Gondzio, F. N. C. Sobral|arXiv (Cornell University)|Jun 26, 2018

Matrix Theory and Algorithms3 citations

TL;DR

This paper proposes a quasi-Newton approach to Interior Point Methods (IPMs) for quadratic programming by approximating the Jacobian of the KKT system using rank-one secant updates, replacing expensive factorizations with cheaper backsolves. The method reduces the number of matrix factorizations—especially effective when factorization cost dominates—demonstrating improved efficiency on quadratic assignment problems.

ABSTRACT

Interior Point Methods (IPM) rely on the Newton method for solving systems of nonlinear equations. Solving the linear systems which arise from this approach is the most computationally expensive task of an interior point iteration. If, due to problem's inner structure, there are special techniques for efficiently solving linear systems, IPMs enjoy fast convergence and are able to solve large scale optimization problems. It is tempting to try to replace the Newton method by quasi-Newton methods. Quasi-Newton approaches to IPMs either are built to approximate the Lagrangian function for nonlinear programming problems or provide an inexpensive preconditioner. In this work we study the impact of using quasi-Newton methods applied directly to the nonlinear system of equations for general quadratic programming problems. The cost of each iteration can be compared to the cost of computing correctors in a usual interior point iteration. Numerical experiments show that the new approach is able to reduce the overall number of matrix factorizations and is suitable for a matrix-free implementation.

Motivation & Objective

• Address the high computational cost of matrix factorizations in interior point methods (IPMs) for large-scale quadratic programs.
• Explore replacing Newton-based linear solves with quasi-Newton updates to reduce the number of expensive factorizations.
• Develop a matrix-free, structure-preserving quasi-Newton method that maintains the sparsity and structure of the unreduced KKT system.
• Investigate whether low-rank Jacobian approximations can match or outperform standard predictor-corrector IPM iterations in terms of factorization count and CPU time.

Proposed method

• Approximate the Jacobian matrix $ J(x, u, z) $ of the KKT system using rank-one secant updates based on function evaluations of $ F $, inspired by the Broyden method.
• Use only the most recent iterate to update the Jacobian approximation, avoiding the need for multiple function evaluations per iteration.
• Preserve the sparsity and block structure of the unreduced KKT matrix to maintain computational efficiency.
• Integrate the quasi-Newton system solve into an IPM framework, replacing the standard corrector step with a quasi-Newton update, with computational cost comparable to one corrector step.
• Implement the method as a modification of the HOPDM solver using the Broyden 'bad' update, adapted to maintain matrix structure.
• Solve the resulting linear system via backsolves after a single factorization, minimizing repeated factorizations.

Experimental results

Research questions

• RQ1 Can a quasi-Newton approach based on rank-one updates reduce the number of matrix factorizations in IPMs for quadratic programs without sacrificing convergence?
• RQ2 Does approximating the Jacobian of the KKT system via function evaluations lead to faster overall solution times when factorization cost dominates?
• RQ3 How well does the low-rank quasi-Newton approximation preserve the structural properties of the unreduced KKT matrix?
• RQ4 In what problem classes does the quasi-Newton IPM outperform standard predictor-corrector IPMs in terms of CPU time and factorization count?
• RQ5 Can the method be effectively implemented in a matrix-free fashion, avoiding repeated factorizations?

Key findings

• The proposed quasi-Newton IPM reduced the number of matrix factorizations in most test problems compared to standard IPM implementations.
• On the QAP test set, the reduction in factorizations was systematically translated into reduced CPU time, indicating the method’s efficiency when factorization cost is high.
• The method achieved comparable performance to standard IPMs with significantly fewer factorizations, particularly benefiting problems where factorization dominates the runtime.
• The Broyden-based update preserved the sparsity and block structure of the unreduced KKT matrix, enabling efficient backsolves.
• The computational cost of solving the quasi-Newton system was comparable to that of computing a single corrector step in standard IPMs.
• The method showed particular promise for problems with high factorization cost relative to backsolve cost, such as large-scale quadratic assignment problems.