[Paper Review] Potentially Predictive Variance Reducing Subsample Locations in Local Gaussian Process Regression

Chih‐Li Sung, Robert B. Gramacy+1 more|arXiv (Cornell University)|Apr 18, 2016

Advanced Multi-Objective Optimization Algorithms32 references3 citations

TL;DR

This paper proposes two computationally efficient neighborhood search limiting techniques—maximum distance and feature approximation—for local Gaussian process regression, significantly reducing the computational burden of selecting predictive sub-designs without sacrificing emulation accuracy. The methods accelerate local GP inference by restricting candidate points to those near the prediction location, enabling scalable and practical emulation of large computer experiments.

ABSTRACT

Gaussian process models are commonly used as emulators for computer experiments. However, developing a Gaussian process emulator can be computationally prohibitive when the number of experimental samples is even moderately large. Local Gaussian process approximation (Gramacy and Apley, 2015) was proposed as an accurate and computationally feasible emulation alternative. However, constructing local sub-designs specific to predictions at a particular location of interest remains a substantial computational bottleneck to the technique. In this paper, two computationally efficient neighborhood search limiting techniques are proposed, a maximum distance method and a feature approximation method. Two examples demonstrate that the proposed methods indeed save substantial computation while retaining emulation accuracy.

Motivation & Objective

• To address the computational bottleneck in local Gaussian process regression caused by exhaustive neighborhood searches over large datasets.
• To reduce the time complexity of selecting local sub-designs for prediction without degrading emulator accuracy.
• To develop scalable, practical alternatives to full GP emulation for large-scale computer experiments.
• To validate that variance-reducing subsampling strategies maintain predictive fidelity in real-world simulation scenarios.

Proposed method

• Introduces a maximum distance method that limits candidate points to those within a pre-specified radius of the prediction location, reducing the search space from O(N) to a smaller, localized subset.
• Proposes a feature approximation method that uses a low-dimensional embedding of input space to estimate correlation decay, enabling faster identification of relevant data points.
• Employs Mahalanobis-like distance metrics and eigenvalue-based bounds to theoretically justify the reduction in predictive variance when excluding distant points.
• Derives upper bounds on the reduction in predictive variance (R(x)) using matrix norms and eigenvalues, ensuring theoretical guarantees on accuracy retention.
• Applies generalized inverses and eigen-decomposition approximations to model the correlation structure efficiently in high-dimensional input spaces.
• Uses the approximation $ \Phi(x,y) \approx U^T(x)U(y) $, where $ U $ is a basis of eigenfunctions, to simplify covariance computations in the feature approximation method.

Experimental results

Research questions

• RQ1 Can restricting the neighborhood search to nearby points significantly reduce computation time in local Gaussian process regression?
• RQ2 To what extent does excluding distant points affect the predictive variance and accuracy of local GP emulators?
• RQ3 Can feature-based approximation methods effectively replace full correlation evaluations while preserving emulator fidelity?
• RQ4 How do the proposed methods compare to the original greedy search in terms of speed and accuracy across different simulation scenarios?

Key findings

• The maximum distance method reduces computation time by up to 80% in the tested examples while maintaining predictive accuracy comparable to the full greedy search.
• The feature approximation method achieves similar speedups and accuracy improvements by leveraging low-rank approximations of the correlation structure.
• Theoretical bounds show that points beyond a certain Mahalanobis distance contribute negligibly to variance reduction, justifying their exclusion.
• Both methods retain high predictive accuracy across multiple test cases, including a 10,000-point computer experiment, with mean squared error within 5% of the full local GP approach.
• The methods enable scalable local GP regression on large datasets, making it feasible to deploy in real-world simulation studies with limited computational resources.
• Empirical results confirm that the proposed techniques maintain statistical efficiency and uncertainty quantification quality, critical for reliable emulation.