[Paper Review] Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)

Andrew Gordon Wilson, Hannes Nickisch|arXiv (Cornell University)|Mar 3, 2015

Gaussian Processes and Bayesian Inference29 references214 citations

TL;DR

This paper introduces the Structured Kernel Interpolation (SKI) framework, a unifying approach that enables scalable Gaussian process inference by approximating kernel matrices through kernel interpolation. Using local cubic interpolation and leveraging Kronecker/Toeplitz structure, KISS-GP achieves O(n) time and storage complexity, enabling high-accuracy, scalable GP inference with arbitrarily located inputs and orders-of-magnitude faster runtime than alternatives like FITC.

ABSTRACT

We introduce a new structured kernel interpolation (SKI) framework, which generalises and unifies inducing point methods for scalable Gaussian processes (GPs). SKI methods produce kernel approximations for fast computations through kernel interpolation. The SKI framework clarifies how the quality of an inducing point approach depends on the number of inducing (aka interpolation) points, interpolation strategy, and GP covariance kernel. SKI also provides a mechanism to create new scalable kernel methods, through choosing different kernel interpolation strategies. Using SKI, with local cubic kernel interpolation, we introduce KISS-GP, which is 1) more scalable than inducing point alternatives, 2) naturally enables Kronecker and Toeplitz algebra for substantial additional gains in scalability, without requiring any grid data, and 3) can be used for fast and expressive kernel learning. KISS-GP costs O(n) time and storage for GP inference. We evaluate KISS-GP for kernel matrix approximation, kernel learning, and natural sound modelling.

Motivation & Objective

• To address the computational bottleneck of standard Gaussian processes, which scale as O(n³) and limit their use to small datasets.
• To unify and generalize existing inducing point methods by framing them as kernel interpolation problems.
• To overcome the grid restrictions of Kronecker and Toeplitz methods, enabling their use with arbitrarily located inputs.
• To enable expressive kernel learning and high-accuracy predictions by efficiently supporting a large number of inducing points.
• To develop a flexible framework—SKI—that allows for the design of new scalable GP methods through customizable interpolation strategies.

Proposed method

• The SKI framework interprets inducing point methods as global GP kernel interpolation to create approximate kernels for scalable inference.
• It introduces local cubic and inverse distance weighting interpolation strategies to approximate the cross-covariance matrix between training and inducing points.
• KISS-GP leverages Kronecker and Toeplitz algebra to achieve O(n + m log m) and O(n + Pm^{1+1/P}) computational complexity, respectively, for P-dimensional inputs.
• The method enables efficient inference with m ≫ n inducing points by exploiting structured matrix algebra, even when inputs are not on a regular grid.
• The framework decomposes the kernel approximation into structured components, allowing for fast matrix operations and reduced storage requirements.
• The approach is implemented as an extension to the GPML toolbox, supporting end-to-end training and prediction with large-scale datasets.

Experimental results

Research questions

• RQ1 How can inducing point methods be unified and generalized through a common framework based on kernel interpolation?
• RQ2 Can kernel interpolation strategies be designed to enable Kronecker and Toeplitz structure to be exploited even with arbitrarily located inputs?
• RQ3 To what extent can increasing the number of inducing points improve predictive accuracy and kernel learning expressivity in scalable GP methods?
• RQ4 How does the choice of interpolation strategy (e.g., local cubic vs. global) affect the trade-off between accuracy and computational efficiency?
• RQ5 Can the SKI framework be used to design new, highly efficient GP models that outperform existing state-of-the-art methods?

Key findings

• KISS-GP achieves O(n) time and storage complexity for GP inference, enabling scalable inference on large datasets.
• For natural sound modeling with 59,306 training points, KISS-GP achieved less than half the standardised mean absolute error (SMAE) of FITC at less than 1% of the runtime.
• With m ∈ [2500, 5000] inducing points, KISS-GP's runtime remained nearly constant, while FITC's runtime increased significantly with m.
• KISS-GP outperformed SSGPR in both accuracy (SMAE ∈ [1.00, 1.05] vs. [1.12, 1.23]) and speed (under 100 seconds vs. up to 8400 seconds) for m ∈ [250, 1250].
• The use of local cubic interpolation in KISS-GP enabled better function curvature recovery than FITC, which over-smoothed due to limited inducing point usage.
• The SKI framework enables the use of m ≫ n inducing points efficiently, unlocking expressive kernel learning and improved predictive performance.