[Paper Review] ADDITIVE COVARIANCE KERNELS FOR HIGH-DIMENSIONAL GAUSSIAN PROCESS MODELING

Nicolas Durrande, David Ginsbourger+1 more|arXiv (Cornell University)|Nov 27, 2011

Advanced Multi-Objective Optimization AlgorithmsComputer Science15 references56 citations

TL;DR

This paper proposes additive covariance kernels for high-dimensional Gaussian process modeling to mitigate the curse of dimensionality. By decomposing the covariance structure into additive components, the method enables scalable, interpretable Kriging models that maintain predictive accuracy while reducing computational cost in high-dimensional input spaces.

ABSTRACT

Gaussian process models -also called Kriging models- are often used as mathematical approximations of expensive experiments. However, the number of observation required for building an emulator becomes unrealistic when using classical covariance kernels when the dimension of input increases. In oder to get round the curse of dimensionality, a popular approach is to consider simplified models such as additive models. The ambition of the present work is to give an insight into covariance kernels that are well suited for building additive Kriging models and to describe some properties of the resulting models.

Motivation & Objective

• Address the computational infeasibility of standard Gaussian process models in high-dimensional input spaces.
• Develop covariance kernels specifically suited for additive Kriging models to improve scalability and interpretability.
• Provide theoretical and practical insights into the properties of additive covariance structures in Gaussian processes.
• Enable efficient emulation of expensive computer experiments with high-dimensional inputs through structured covariance decomposition.

Proposed method

• Propose a class of additive covariance kernels that decompose the total covariance into independent contributions from each input dimension.
• Construct the kernel as a sum of univariate covariance functions, each acting on a single input variable.
• Ensure the resulting kernel remains positive definite and suitable for Gaussian process regression.
• Utilize the additive structure to reduce the number of hyperparameters and computational complexity compared to full-rank kernels.
• Enable efficient inference and prediction by exploiting conditional independence structures induced by the additive decomposition.
• Apply the framework to computer experiments and emulation tasks where input dimensionality is high but the underlying function may be approximately additive.

Experimental results

Research questions

• RQ1 How can covariance kernels be structured to support efficient and interpretable Gaussian process modeling in high-dimensional input spaces?
• RQ2 What are the theoretical and practical properties of additive covariance kernels in the context of Kriging?
• RQ3 To what extent does the additive kernel structure reduce computational cost while preserving predictive accuracy?
• RQ4 Can additive kernels effectively model functions that are approximately additive, even when the true function is not strictly so?
• RQ5 How does the additive kernel framework compare to standard full-rank covariance kernels in terms of scalability and model interpretability?

Key findings

• Additive covariance kernels significantly reduce the number of hyperparameters required compared to full-rank kernels, improving model scalability.
• The additive structure enables efficient computation of posterior predictions and marginal likelihoods, even in high-dimensional settings.
• The resulting models maintain good predictive performance on functions that are approximately additive, despite simplifying assumptions.
• The framework supports interpretable modeling by allowing individual input dimensions to contribute independently to the overall covariance structure.
• Theoretical properties such as positive definiteness are preserved under mild conditions on the univariate kernel components.
• Empirical results demonstrate that additive kernels outperform standard kernels in terms of computational efficiency without substantial loss in accuracy on high-dimensional test functions.