[Paper Review] An analytic comparison of regularization methods for Gaussian Processes

Hossein Mohammadi, Rodolphe Le Riche+3 more|arXiv (Cornell University)|Feb 2, 2016

Advanced Multi-Objective Optimization Algorithms20 references18 citations

TL;DR

This paper provides a rigorous analytic comparison of pseudoinverse (PI) and nugget regularization in Gaussian Processes (GPs), showing that PI averages responses at redundant points and sets variance to zero, while nugget regularization preserves individual values. The key contribution is a data-model discrepancy measure to guide regularization choice, along with a novel distribution-wise GP that improves robustness by interpolating distributions instead of point values.

ABSTRACT

Gaussian Processes (GPs) are a popular approach to predict the output of a parameterized experiment. They have many applications in the field of Computer Experiments, in particular to perform sensitivity analysis, adaptive design of experiments and global optimization. Nearly all of the applications of GPs require the inversion of a covariance matrix that, in practice, is often ill-conditioned. Regularization methodologies are then employed with consequences on the GPs that need to be better understood.The two principal methods to deal with ill-conditioned covariance matrices are i) pseudoinverse and ii) adding a positive constant to the diagonal (the so-called nugget regularization).The first part of this paper provides an algebraic comparison of PI and nugget regularizations. Redundant points, responsible for covariance matrix singularity, are defined. It is proven that pseudoinverse regularization, contrarily to nugget regularization, averages the output values and makes the variance zero at redundant points. However, pseudoinverse and nugget regularizations become equivalent as the nugget value vanishes. A measure for data-model discrepancy is proposed which serves for choosing a regularization technique.In the second part of the paper, a distribution-wise GP is introduced that interpolates Gaussian distributions instead of data points. Distribution-wise GP can be seen as an improved regularization method for GPs.

Motivation & Objective

• To analytically compare pseudoinverse (PI) and nugget regularization in Gaussian Processes for handling ill-conditioned covariance matrices.
• To define and characterize redundant points that cause covariance matrix singularity in GP regression.
• To develop a data-model discrepancy measure to guide the selection between PI and nugget regularization.
• To introduce a distribution-wise GP framework that interpolates probability distributions rather than point observations, improving regularization robustness.
• To establish theoretical equivalence of PI and nugget regularization in the limit as the nugget value approaches zero.

Proposed method

• Uses eigenvalue decomposition of the covariance matrix C to analyze the effects of PI and nugget regularization on the solution space.
• Defines redundant points as those with identical input coordinates, leading to zero eigenvalues in the covariance matrix.
• Derives the likelihood function for GP regression under both regularization schemes using the eigen-decomposition of C.
• Introduces a discrepancy measure Δ(τ²) that quantifies the difference in log-likelihood between original and averaged response data.
• Proposes a distribution-wise GP model that replaces point observations with empirical distributions of repeated responses.
• Establishes that the maximum likelihood estimate of the nugget τ² is non-decreasing when response variance is increased at redundant points, proving the monotonicity of the discrepancy measure.

Experimental results

Research questions

• RQ1 How do pseudoinverse and nugget regularization differ in their treatment of redundant data points in Gaussian Process regression?
• RQ2 What is the theoretical relationship between pseudoinverse and nugget regularization, and under what conditions are they equivalent?
• RQ3 Can a data-model discrepancy measure be derived to objectively select between PI and nugget regularization?
• RQ4 How does the distribution-wise GP model improve upon standard GP regularization in the presence of repeated observations?
• RQ5 Does increasing the variance at redundant points lead to a larger estimated nugget value in maximum likelihood estimation?

Key findings

• Pseudoinverse regularization averages response values at redundant points and sets the kriging variance to zero at those locations, while nugget regularization preserves individual values.
• As the nugget value τ² approaches zero, pseudoinverse and nugget regularization become asymptotically equivalent in their predictive mean and variance.
• The data-model discrepancy measure Δ(τ²) is monotonically decreasing and quantifies the log-likelihood difference between original and averaged response data.
• The maximum likelihood estimate of the nugget τ² is never smaller when response variance is increased at redundant points, and strictly larger if the variance increase is positive.
• The distribution-wise GP model, which interpolates distributions of responses at repeated points, provides a more robust regularization framework than standard point-based GP.
• Theoretical analysis confirms that the likelihood-based selection of τ² is consistent with the observed variability at redundant points, validating the discrepancy measure as a decision criterion.