[Paper Review] Additive Gaussian Process Regression

Shaan Qamar, Surya T. Tokdar|arXiv (Cornell University)|Nov 25, 2014

Gaussian Processes and Bayesian Inference33 references8 citations

TL;DR

This paper proposes a Bayesian additive Gaussian process (AGP) regression model that enables efficient high-dimensional nonparametric regression by decomposing the regression function into additive components of low-dimensional, nonparametric functions. Using a novel Markov chain Monte Carlo sampler with multiple-try Metropolis moves and stochastic search, the method achieves state-of-the-art support and interaction recovery while significantly improving prediction accuracy over competitors in both simulated and real-world data.

ABSTRACT

Additive-interactive regression has recently been shown to offer attractive minimax error rates over traditional nonparametric multivariate regression in a wide variety of settings, including cases where the predictor count is much larger than the sample size and many of the predictors have important effects on the response, potentially through complex interactions. We present a Bayesian implementation of additive-interactive regression using an additive Gaussian process (AGP) prior and develop an efficient Markov chain sampler that extends stochastic search variable selection in this setting. Careful prior and hyper-parameter specification are developed in light of performance and computational considerations, and key innovations address difficulties in exploring a joint posterior distribution over multiple subsets of high dimensional predictor inclusion vectors. The method offers state-of-the-art support and interaction recovery while improving dramatically over competitors in terms of prediction accuracy on a diverse set of simulated and real data. Results from real data studies provide strong evidence that the additive-interactive framework is an attractive modeling platform for high-dimensional nonparametric regression.

Motivation & Objective

• To develop a fully Bayesian framework for additive-interactive nonparametric regression in high-dimensional settings where $ p \gg n $.
• To overcome the curse of dimensionality in traditional nonparametric regression by modeling the regression function as a sum of low-dimensional component functions.
• To enable accurate recovery of predictor importance and interaction structures through a structured prior and efficient posterior exploration.
• To improve computational scalability and statistical performance over existing methods in high-dimensional, sparse, and smooth regression settings.

Proposed method

• Employs an additive Gaussian process (AGP) prior where each component function depends on a small subset of predictors, enabling flexible, nonparametric modeling of complex interactions.
• Uses a novel Markov chain Monte Carlo (MCMC) sampler with three types of moves: cross-donate (CD), paired-donate (PD), and paired-swap (PS) to explore high-dimensional predictor inclusion vectors.
• Incorporates multiple-try Metropolis proposals to enhance mixing and convergence in the joint posterior over component structures and predictor assignments.
• Implements stochastic search variable selection via component-wise moves that dynamically reassign predictors across additive components to explore model space efficiently.
• Applies careful prior and hyperparameter specification to balance model flexibility, computational cost, and estimation accuracy.
• Uses a mixture of ICM proposals with symmetric transition kernels to preserve detailed balance and ensure stationarity in the MCMC chain.

Experimental results

Research questions

• RQ1 Can a Bayesian additive Gaussian process model achieve superior prediction accuracy and variable selection in high-dimensional, nonparametric regression with complex interactions?
• RQ2 How effectively can the proposed MCMC sampler explore the high-dimensional space of predictor inclusion vectors and component structures?
• RQ3 To what extent does the additive-interactive framework outperform traditional nonparametric and parametric methods in terms of support recovery and interaction detection?
• RQ4 Does the method maintain good performance under extreme sparsity and high $ p/n $ ratios, as predicted by minimax theory?

Key findings

• The AGP model achieves state-of-the-art support and interaction recovery, significantly outperforming competitors in identifying relevant predictors and their interactions.
• Prediction accuracy is dramatically improved over existing methods, including BART, random forests, and the Lasso, especially in high-dimensional and smooth regression settings.
• The method successfully recovers true interaction patterns in simulated data, with inclusion probabilities closely tracking the ground-truth predictor importance.
• Real data studies on Boston housing, crime, riboflavin, and cookie datasets demonstrate strong empirical performance and robustness across diverse applications.
• The MCMC sampler achieves good mixing and convergence, with trace plots of $ \sigma^2 $ showing stable posterior exploration and reliable inference.
• Theoretical and empirical results confirm that the additive-interactive framework breaks the extreme sparsity assumption of traditional nonparametric models, enabling effective learning in high-dimensional settings.