[Paper Review] Scaling Gaussian Process Regression with Derivatives

David Eriksson, Kun Dong+3 more|arXiv (Cornell University)|Jan 1, 2018

Gaussian Processes and Bayesian InferenceComputer Science29 citations

TL;DR

This paper proposes scalable Gaussian process regression with derivatives using iterative solvers and pivoted Cholesky preconditioning, reducing computational complexity from 𝒪(n³d³) to near-linear scaling. The method enables efficient Bayesian optimization and large-scale problems in high dimensions by accelerating matrix operations via fast matrix-vector multiplications and effective preconditioning.

ABSTRACT

Gaussian processes (GPs) with derivatives are useful in many applications, including Bayesian optimization, implicit surface reconstruction, and terrain reconstruction. Fitting a GP to function values and derivatives at $n$ points in $d$ dimensions requires linear solves and log determinants with an ${n(d+1) imes n(d+1)}$ positive definite matrix-- leading to prohibitive $\mathcal{O}(n^3d^3)$ computations for standard direct methods. We propose iterative solvers using fast $\mathcal{O}(nd)$ matrix-vector multiplications (MVMs), together with pivoted Cholesky preconditioning that cuts the iterations to convergence by several orders of magnitude, allowing for fast kernel learning and prediction. Our approaches, together with dimensionality reduction, allows us to scale Bayesian optimization with derivatives to high-dimensional problems and large evaluation budgets.

Motivation & Objective

• To address the prohibitive computational cost of fitting Gaussian processes to function values and derivatives in high-dimensional settings.
• To reduce the O(n³d³) complexity of standard direct methods for solving GP regression with derivatives.
• To enable scalable Bayesian optimization and large evaluation budgets using derivative information.
• To develop efficient iterative solvers with fast matrix-vector multiplications and effective preconditioning for large-scale kernel learning.

Proposed method

• Utilizes iterative Krylov subspace solvers (e.g., conjugate gradient) for solving linear systems arising in GP regression with derivatives.
• Employs fast O(nd) matrix-vector multiplications to accelerate kernel evaluations and linear solves.
• Applies pivoted Cholesky preconditioning to the n(d+1) × n(d+1) covariance matrix to drastically reduce the number of iterations to convergence.
• Combines the iterative solver with pivoted Cholesky to achieve near-optimal convergence rates for large-scale GP inference.
• Integrates dimensionality reduction techniques to further scale the method to high-dimensional input spaces.

Experimental results

Research questions

• RQ1 Can iterative solvers with fast matrix-vector multiplications significantly reduce the computational burden of GP regression with derivatives?
• RQ2 To what extent does pivoted Cholesky preconditioning accelerate convergence in derivative-informed GP regression?
• RQ3 Can the proposed method scale Bayesian optimization with derivatives to high-dimensional problems and large evaluation budgets?
• RQ4 How does the combination of iterative solvers and preconditioning compare to standard direct methods in terms of scalability and accuracy?

Key findings

• The proposed method reduces the computational complexity of GP regression with derivatives from O(n³d³) to near-linear scaling with respect to n and d.
• Pivoted Cholesky preconditioning reduces the number of conjugate gradient iterations by several orders of magnitude, enabling fast convergence.
• The approach supports scalable Bayesian optimization with derivatives on high-dimensional problems and large evaluation budgets.
• Fast matrix-vector multiplications enable efficient kernel learning and prediction, making large-scale derivative-informed GP regression feasible.