[Paper Review] Additive Gaussian Processes

David Duvenaud, Hannes Nickisch+1 more|arXiv (Cornell University)|Dec 19, 2011

Gaussian Processes and Bayesian Inference15 references203 citations

TL;DR

This paper introduces Additive Gaussian Processes (AGP), a novel GP model that generalizes both Generalized Additive Models (GAMs) and standard squared-exponential GP models by enabling flexible, structured additive interactions of all orders. The method uses a parameterization that efficiently computes an exponential number of interaction terms with only O(D) hyperparameters, achieving state-of-the-art predictive performance and improved interpretability through learned interaction order variances.

ABSTRACT

We introduce a Gaussian process model of functions which are additive. An additive function is one which decomposes into a sum of low-dimensional functions, each depending on only a subset of the input variables. Additive GPs generalize both Generalized Additive Models, and the standard GP models which use squared-exponential kernels. Hyperparameter learning in this model can be seen as Bayesian Hierarchical Kernel Learning (HKL). We introduce an expressive but tractable parameterization of the kernel function, which allows efficient evaluation of all input interaction terms, whose number is exponential in the input dimension. The additional structure discoverable by this model results in increased interpretability, as well as state-of-the-art predictive power in regression tasks.

Motivation & Objective

• To develop a Gaussian process model that captures low- and high-order input interactions in a structured, interpretable way.
• To generalize both GAMs (first-order interactions) and standard SE-GPs (full-order interactions) into a single unified framework.
• To enable efficient learning of kernel hyperparameters and interaction order variances despite the exponential number of interaction terms.
• To improve model interpretability by identifying which orders of interaction are important in real datasets.
• To outperform existing methods like HKL and standard SE-GP in predictive accuracy while maintaining computational tractability.

Proposed method

• The model uses a kernel function composed of additive terms across all possible interaction orders, from first-order to D-th-order interactions.
• Each interaction order is parameterized by a variance hyperparameter σ²ₙ and a product of one-dimensional base kernels kᵢ(xᵢ, x′ᵢ).
• The kernel is efficiently computed via a recursive decomposition that avoids explicitly summing over all D choose n terms.
• Hyperparameter learning is performed via marginal likelihood maximization, enabling Bayesian hierarchical kernel learning (HKL).
• The model supports both regression and classification via GP inference with EP or Laplace approximation.
• A constant mean function is fitted alongside the kernel, and the method is implemented with efficient code for practical use.

Experimental results

Research questions

• RQ1 Can a Gaussian process model effectively capture structured additive interactions of all orders while remaining computationally tractable?
• RQ2 Does the ability to learn interaction order variances improve predictive performance compared to standard GP models?
• RQ3 How does the proposed model compare to Hierarchical Kernel Learning (HKL) in terms of predictive accuracy and hyperparameter learning?
• RQ4 To what extent does the additive structure enhance model interpretability in real-world regression tasks?
• RQ5 Can the model generalize well to unseen input combinations by learning low-order interaction patterns?

Key findings

• The Additive GP achieved state-of-the-art performance on regression tasks, outperforming both GP-GAM and SE-GP models on datasets with low-order interaction structures.
• On the pumadyn-8nh and housing datasets, the Additive GP achieved the lowest mean squared error (0.316 and 0.102, respectively), significantly outperforming GP-GAM and HKL.
• In negative log-likelihood comparisons, the Additive GP matched or exceeded the performance of SE-GP and GP-GAM across all datasets, with the lowest NLL on bach and housing.
• The model never performed significantly worse than any other method, and in some cases significantly better, indicating robustness and strong generalization.
• On classification tasks, the Additive GP achieved the lowest percent error on the breast and sonar datasets, and the lowest NLL on all but one dataset, showing strong discriminative performance.
• The model's performance was competitive with HKL but superior in predictive accuracy, and it avoided HKL’s reliance on cross-validation for hyperparameter tuning.