[Paper Review] Product Kernel Interpolation for Scalable Gaussian Processes

Jacob R. Gardner, Geoff Pleiss+3 more|arXiv (Cornell University)|Feb 24, 2018

Gaussian Processes and Bayesian Inference30 references21 citations

TL;DR

This paper introduces SKIP (Product Kernel Interpolation for Scalable Gaussian Processes), a method that enables efficient matrix-vector multiplications (MVMs) with product kernels by leveraging Lanczos decompositions of individual kernel components. The approach reduces computational complexity from exponential to linear in dimensionality, enabling scalable GP inference for high-dimensional and multi-task learning with state-of-the-art asymptotic complexity.

ABSTRACT

Recent work shows that inference for Gaussian processes can be performed efficiently using iterative methods that rely only on matrix-vector multiplications (MVMs). Structured Kernel Interpolation (SKI) exploits these techniques by deriving approximate kernels with very fast MVMs. Unfortunately, such strategies suffer badly from the curse of dimensionality. We develop a new technique for MVM based learning that exploits product kernel structure. We demonstrate that this technique is broadly applicable, resulting in linear rather than exponential runtime with dimension for SKI, as well as state-of-the-art asymptotic complexity for multi-task GPs.

Motivation & Objective

• To address the curse of dimensionality in structured kernel interpolation (SKI), which suffers from exponential growth in inducing point requirements with input dimension.
• To improve modularity in MVM-based GP inference by enabling efficient computation of MVMs with element-wise product kernels, even when components use different grids.
• To achieve linear runtime scaling with dimension for product kernel GP inference, overcoming the exponential complexity of standard SKI.
• To enable state-of-the-art asymptotic complexity for multi-task Gaussian processes (MTGPs), reducing inference to O(n + m log m + s) for s tasks.
• To develop a practical, GPU-accelerated implementation in GPyTorch for broad applicability in real-world GP applications.

Proposed method

• The method uses Lanczos decomposition to approximate MVMs with each component kernel in a product kernel, enabling fast computation of the full product kernel MVM.
• It combines the Lanczos approximations of individual kernels through a merge step that maintains computational efficiency, with complexity O(drμ(K) + r³n log d).
• SKIP enables structured kernel interpolation (SKI) with product kernels without relying on Kronecker factorizations, avoiding exponential growth in m with dimension.
• For high-dimensional regression, the method decomposes the d-dimensional kernel into d one-dimensional kernels, achieving O(dn + dm log m) complexity instead of O(n + dm^d log m).
• For multi-task GP, SKIP reduces inference complexity to O(n + m log m + s) by exploiting product structure in the kernel and enabling fast MVMs with the full kernel matrix.
• The approach supports Gibbs sampling for task clustering by efficiently computing MVMs with the product kernel, enabling scalable discovery of task groupings.

Experimental results

Research questions

• RQ1 Can MVMs with product kernels be computed efficiently without relying on Kronecker factorizations, which suffer from exponential scaling in dimension?
• RQ2 Can the modularity of MVM-based GP inference be preserved when kernels are composed as element-wise products, even when components use different grids?
• RQ3 Does exploiting product kernel structure enable linear scaling with dimension in GP inference, overcoming the curse of dimensionality in SKI?
• RQ4 Can SKIP achieve state-of-the-art asymptotic complexity for multi-task Gaussian processes while maintaining practical efficiency?
• RQ5 Can the method be efficiently implemented on GPUs to enable scalable inference in real-world applications?

Key findings

• SKIP enables MVMs with product kernels in O(drμ(K) + r³n log d) time, where r is a small constant, making it feasible for high-dimensional problems.
• For d-dimensional regression, the method reduces complexity from O(n + dm^d log m) in standard SKI to O(dn + dm log m), achieving linear scaling in dimension.
• In multi-task GP settings, SKIP reduces inference complexity to O(n + m log m + s), where s is the number of tasks, outperforming prior methods in asymptotic complexity.
• The method enables scalable Gibbs sampling for task clustering by efficiently computing MVMs with the product kernel, allowing discovery of meaningful task groupings.
• Empirical results show the cluster-based MTGP model outperforms both single-task and standard MTGP baselines in extrapolation accuracy on pediatric development data.
• The authors release a GPU-accelerated implementation in GPyTorch, demonstrating practical scalability and usability for real-world applications.