[Paper Review] Kernel Interpolation for Scalable Online Gaussian Processes

Samuel Stanton, Wesley J. Maddox+2 more|arXiv (Cornell University)|Mar 2, 2021

Gaussian Processes and Bayesian Inference35 references4 citations

TL;DR

This paper proposes WISKI (Woodbury Inversion with SKI), a method for constant-time $Ó(1)$ online updates in Gaussian processes using structured kernel interpolation and the Woodbury identity, enabling exact inference with $Ó(1)$ computational cost per update. It achieves state-of-the-art speed comparable to variational methods while maintaining calibrated uncertainty and exact posterior inference, validated across online regression, Bayesian optimization, and active learning for malaria forecasting.

ABSTRACT

Gaussian processes (GPs) provide a gold standard for performance in online settings, such as sample-efficient control and black box optimization, where we need to update a posterior distribution as we acquire data in a sequential fashion. However, updating a GP posterior to accommodate even a single new observation after having observed $n$ points incurs at least $O(n)$ computations in the exact setting. We show how to use structured kernel interpolation to efficiently recycle computations for constant-time $O(1)$ online updates with respect to the number of points $n$, while retaining exact inference. We demonstrate the promise of our approach in a range of online regression and classification settings, Bayesian optimization, and active sampling to reduce error in malaria incidence forecasting. Code is available at https://github.com/wjmaddox/online_gp.

Motivation & Objective

• To address the high computational cost of exact online Gaussian process updates, which scale as $Ó(n^3)$ for $n$ data points.
• To enable exact inference in online settings without the approximation errors common in sparse variational methods.
• To achieve $Ó(1)$ update complexity with respect to $n$, while maintaining fast test-time predictions and calibrated uncertainty.
• To support real-time decision-making in applications like Bayesian optimization and active sampling, where uncertainty calibration is critical.
• To provide a scalable, exact, and efficient alternative to existing online GP methods that rely on low-rank approximations or variational inference.

Proposed method

• Leverages structured kernel interpolation (SKI) to approximate the kernel matrix using a fixed set of inducing points, enabling efficient kernel evaluation.
• Applies the Woodbury identity to efficiently update the predictive distribution and marginal likelihood after each new observation, avoiding recomputation from scratch.
• Caches intermediate computations (e.g., inverse of the kernel matrix over inducing points) to recycle them across online updates.
• Reuses the same inducing point structure across updates, ensuring that the computational cost per new point remains constant at $Ó(1)$ in $n$.
• Maintains exact inference by avoiding variational approximations, preserving calibrated uncertainty estimates.
• Uses a batched upper confidence bound (UCB) acquisition function in Bayesian optimization, with online updates via WISKI to maintain high performance.

Experimental results

Research questions

• RQ1 Can we achieve $Ó(1)$ online updates for Gaussian processes while retaining exact inference and calibrated uncertainty?
• RQ2 How does WISKI compare to variational and exact GP methods in terms of speed, accuracy, and uncertainty calibration in online regression tasks?
• RQ3 Can WISKI effectively support active learning and Bayesian optimization with fast, exact updates and global variance reduction?
• RQ4 Does WISKI avoid the overfitting and poor generalization issues seen in streaming variational GP methods like O-SVGP?
• RQ5 Can WISKI scale to large online datasets with minimal computational overhead while maintaining predictive performance?

Key findings

• WISKI achieves $Ó(1)$ update complexity with respect to $n$, matching the speed of variational methods while retaining exact inference.
• In online regression on exchange rate data, WISKI outperforms O-SVGP and O-SGPR by learning high-frequency trends and avoiding overfitting to early data.
• On the UCI Power Plant dataset, WISKI matches the performance of exact GPs and outperforms O-SVGP, which overestimates noise and converges to a suboptimal solution.
• In Bayesian optimization on Ackley and Levy functions, WISKI is significantly faster than exact GP and O-SVGP, while achieving comparable or better optimization performance.
• In active learning for malaria incidence forecasting, WISKI continues to reduce RMSE over 500 iterations, while O-SVGP stagnates after ~250 iterations due to local clustering of query points.
• WISKI selects diverse, globally representative query points in active learning, reducing global variance more effectively than O-SVGP, which tends to cluster points in dense regions.