[Paper Review] Randomly Projected Additive Gaussian Processes for Regression

Ian Delbridge, David Bindel+1 more|arXiv (Cornell University)|Dec 30, 2019

Gaussian Processes and Bayesian Inference37 references12 citations

TL;DR

This paper proposes Randomly Projected Additive Gaussian Processes (RPA-GP) and Deterministic Projected Additive GP (DPA-GP), which use sums of one-dimensional kernels applied to random or deterministic projections of high-dimensional inputs. Surprisingly, performance converges rapidly to that of full-dimensional kernels—even with single-dimensional projections—enabling fast, scalable regression without learning projections.

ABSTRACT

Gaussian processes (GPs) provide flexible distributions over functions, with inductive biases controlled by a kernel. However, in many applications Gaussian processes can struggle with even moderate input dimensionality. Learning a low dimensional projection can help alleviate this curse of dimensionality, but introduces many trainable hyperparameters, which can be cumbersome, especially in the small data regime. We use additive sums of kernels for GP regression, where each kernel operates on a different random projection of its inputs. Surprisingly, we find that as the number of random projections increases, the predictive performance of this approach quickly converges to the performance of a kernel operating on the original full dimensional inputs, over a wide range of data sets, even if we are projecting into a single dimension. As a consequence, many problems can remarkably be reduced to one dimensional input spaces, without learning a transformation. We prove this convergence and its rate, and additionally propose a deterministic approach that converges more quickly than purely random projections. Moreover, we demonstrate our approach can achieve faster inference and improved predictive accuracy for high-dimensional inputs compared to kernels in the original input space.

Motivation & Objective

• To address the curse of dimensionality in Gaussian process regression without requiring trainable projections.
• To develop a learning-free method that maintains predictive accuracy while drastically reducing computational complexity.
• To demonstrate that additive kernels over random projections can approximate full-dimensional kernels with minimal projections.
• To enable efficient inference using structured kernel interpolation (SKI) by exploiting additive structure in projected inputs.
• To compare the performance of RPA-GP and DPA-GP against standard GP kernels and additive models on high-dimensional and large-scale datasets.

Proposed method

• RPA-GP constructs additive GPs by applying one-dimensional kernels to multiple random projections of the input data.
• Each kernel operates independently on a random projection, and their sum forms the overall kernel function.
• The method leverages the Turning Band Method (TBM) to approximate high-dimensional kernels through additive projections.
• DPA-GP introduces a deterministic projection scheme that minimizes redundancy and accelerates convergence to the limiting kernel.
• The approach enables the use of Structured Kernel Interpolation (SKI) by decomposing the problem into J one-dimensional subproblems.
• Theoretical analysis proves that RPA-GP converges to an inverse multiquadratic kernel at rate O(J−1/2) as the number of projections J increases.

Experimental results

Research questions

• RQ1 Can random projections into low-dimensional spaces achieve predictive performance comparable to full-dimensional GP kernels without learning the projections?
• RQ2 How quickly does the predictive performance of RPA-GP converge to that of a full-dimensional kernel as the number of projections increases?
• RQ3 Can a deterministic projection scheme (DPA-GP) outperform purely random projections in convergence speed and accuracy?
• RQ4 Does exploiting additive structure in projected inputs enable scalable inference via SKI in high-dimensional settings?
• RQ5 How does RPA-GP compare to standard GP kernels and additive models (e.g., GAM GP) in terms of accuracy and scalability on large or high-dimensional datasets?

Key findings

• RPA-GP achieves predictive performance comparable to full-dimensional kernels even when projecting into a single dimension, with convergence observed across diverse datasets.
• The convergence rate of RPA-GP to the inverse multiquadratic kernel is proven to be O(J−1/2), where J is the number of projections.
• DPA-GP, a deterministic projection method, converges faster than RPA-GP and improves predictive accuracy, especially on large datasets.
• RPA-GP with SKI enables linear-time training and constant-time prediction, reducing computational complexity from O(n³) to O(Jc(n + m)) with m inducing points.
• On high-dimensional data (e.g., d=1935), DPA-GP-ARD outperforms RBF and GAM GPs when the number of training points n is small relative to input dimension d.
• Empirical results show that RPA-GP and DPA-GP scale effectively to over 1000 input dimensions, with training time scaling linearly with n, unlike Cholesky-based inference.