[Paper Review] Krylov Subspace Recycling for Fast Iterative Least-Squares in Machine Learning

Filip de Roos, Philipp Hennig|arXiv (Cornell University)|Jun 1, 2017

Gaussian Processes and Bayesian Inference18 references3 citations

TL;DR

This paper proposes Krylov subspace recycling to accelerate iterative least-squares solvers in machine learning by reusing information across sequences of related symmetric positive definite linear systems. By adapting low-rank subspaces over time at low additional cost, the method achieves near-exact solutions with computational cost scaling quadratically in data size, outperforming fixed low-rank approximations in precision while remaining scalable to 10⁵–10⁶ data points.

ABSTRACT

Solving symmetric positive definite linear problems is a fundamental computational task in machine learning. The exact solution, famously, is cubicly expensive in the size of the matrix. To alleviate this problem, several linear-time approximations, such as spectral and inducing-point methods, have been suggested and are now in wide use. These are low-rank approximations that choose the low-rank space a priori and do not refine it over time. While this allows linear cost in the data-set size, it also causes a finite, uncorrected approximation error. Authors from numerical linear algebra have explored ways to iteratively refine such low-rank approximations, at a cost of a small number of matrix-vector multiplications. This idea is particularly interesting in the many situations in machine learning where one has to solve a sequence of related symmetric positive definite linear problems. From the machine learning perspective, such deflation methods can be interpreted as transfer learning of a low-rank approximation across a time-series of numerical tasks. We study the use of such methods for our field. Our empirical results show that, on regression and classification problems of intermediate size, this approach can interpolate between low computational cost and numerical precision.

Motivation & Objective

• To address the high computational cost of exact solutions to symmetric positive definite linear systems in machine learning, which scale cubically with data size.
• To explore whether iterative solvers can be made adaptive across sequences of related problems by reusing information from prior solves.
• To evaluate if Krylov subspace recycling offers a practical middle ground between low-cost approximate methods and exact solvers in terms of speed and accuracy.
• To compare the performance of recycling-based iterative solvers against standard inducing-point and spectral approximation methods in real ML workloads.

Proposed method

• The method employs deflated conjugate gradients (def-CG), an iterative Krylov subspace method that reuses Ritz vectors from previous solves to accelerate convergence on subsequent systems.
• It maintains a low-dimensional subspace of search directions derived from previous solutions, which are deflated from the Krylov iteration to reduce the effective condition number.
• The algorithm initializes with a standard CG solve on the first system, then uses the computed Ritz vectors as a deflation subspace for later systems.
• The deflation subspace is updated incrementally across iterations, allowing progressive refinement of the low-rank approximation without full recomputation.
• The approach is applied to sequences of linear systems arising in Bayesian logistic regression and Gaussian process classification, where the Hessian or kernel matrix changes gradually.
• The method is compared to standard CG and fixed low-rank approximations (e.g., inducing points) using CPU time and relative residual error as metrics.

Experimental results

Research questions

• RQ1 Can Krylov subspace recycling significantly reduce the time to solve sequences of related symmetric positive definite linear systems in machine learning?
• RQ2 How does the accuracy of recycling-based iterative solvers compare to fixed low-rank approximation methods like inducing points?
• RQ3 To what extent does reusing information from prior solves reduce the number of iterations and effective condition number in subsequent systems?
• RQ4 Can recycling-based solvers achieve high-precision solutions at a computational cost that remains competitive with linear-cost approximations?

Key findings

• Deflated conjugate gradients (def-CG) achieved a 6-order-of-magnitude improvement in precision over inducing-point methods while incurring only a modest increase in computational cost.
• For data sets of size ~10⁵ to ~10⁶, the method remained computationally viable, with runtimes scaling quadratically with data size.
• The relative residual decreased more rapidly across Newton iterations, confirming that recycled subspaces lower the effective condition number and accelerate convergence.
• At comparable computational cost, def-CG outperformed inducing-point methods using 25–50% of the data in terms of log-likelihood accuracy.
• The method demonstrated consistent performance gains across multiple regression and classification tasks, particularly in later iterations of optimization loops.
• The approach provides a practical trade-off between computational cost and numerical precision, bridging the gap between exact solvers and fixed low-rank approximations.