[Paper Review] Exact Inference for Gaussian Process Regression in case of Big Data with the Cartesian Product Structure

Mikhail Belyaev, Evgeny Burnaev+1 more|arXiv (Cornell University)|Mar 26, 2014

Gaussian Processes and Bayesian Inference13 references20 citations

TL;DR

This paper proposes a novel exact inference method for Gaussian Process regression tailored to large-scale factorial designs with multidimensional factors. By exploiting the Cartesian product structure of the data, the method enables efficient, accurate, and anisotropy-aware GP regression with O(N log N) time complexity, outperforming sparse approximations like FITC on both accuracy and runtime in real-world and synthetic benchmarks.

ABSTRACT

Approximation algorithms are widely used in many engineering problems. To obtain a data set for approximation a factorial design of experiments is often used. In such case the size of the data set can be very large. Therefore, one of the most popular algorithms for approximation - Gaussian Process regression - can be hardly applied due to its computational complexity. In this paper a new approach for Gaussian Process regression in case of factorial design of experiments is proposed. It allows to efficiently compute exact inference and handle large multidimensional data sets. The proposed algorithm provides fast and accurate approximation and also handles anisotropic data.

Motivation & Objective

• To address the computational intractability of standard Gaussian Process regression on large multidimensional datasets arising from factorial designs.
• To develop an exact inference method that preserves the full information in large training sets without relying on sparse approximations.
• To handle anisotropic data—where factor sizes vary significantly—by incorporating a tailored prior distribution that prevents model degeneracy.
• To enable efficient, scalable GP regression that maintains the flexibility and interpretability of GP models on high-dimensional, structured data.
• To demonstrate the method’s superiority over existing approaches like FITC and MARS in accuracy and runtime on both synthetic and real-world engineering problems.

Proposed method

• The method exploits the Cartesian product structure of factorial designs to decompose the covariance matrix into a tensor product of smaller, structured matrices.
• It uses a low-rank approximation of the covariance matrix based on the Kronecker product structure, enabling efficient Cholesky decomposition via FFT-based solvers.
• A novel prior distribution is introduced that accounts for varying factor sizes, improving robustness to anisotropy and preventing model degeneracy.
• The algorithm performs exact inference using fast matrix operations, achieving O(N log N) time complexity and O(N) memory usage.
• Parameter initialization is guided by domain knowledge of factor sizes, enhancing convergence and stability in anisotropic settings.
• The approach is extended to include regularization that adapts to the intrinsic structure of the design, improving generalization.

Experimental results

Research questions

• RQ1 Can exact Gaussian Process inference be made computationally feasible for large multidimensional factorial designs?
• RQ2 How can the Cartesian product structure of factorial designs be leveraged to accelerate GP regression without approximation?
• RQ3 What prior distribution and initialization strategy best handle anisotropic data with varying factor sizes?
• RQ4 How does the proposed method compare in accuracy and efficiency to sparse GP approximations like FITC and MARS on structured data?
• RQ5 Can the method maintain high predictive accuracy while scaling to datasets with tens of thousands of points?

Key findings

• The proposed tensorGP-reg algorithm achieves significantly better approximation accuracy than FITC and MARS across a range of test problems, as shown by Dolan-Moré curves in logarithmic scale.
• On the rotating disc design problem with 14,400 training points, tensorGP-reg produced smooth, accurate approximations, while FITC with 500 inducing points exhibited degeneracy and poor fit.
• The method achieves O(N log N) time complexity and O(N) memory usage, enabling exact inference on large datasets where standard GP regression is infeasible.
• The tailored prior distribution effectively mitigates model degeneracy in anisotropic data, outperforming standard GP formulations on high-dimensional, unevenly sampled designs.
• TensorGP-reg's runtime performance is comparable to MARS and superior to FITC, with Dolan-Moré curves showing consistent advantages in both accuracy and speed.
• The algorithm successfully preserves the full information in the training set, avoiding the information loss inherent in sparse approximations.