[Paper Review] Extended T-process Regression Models

Zhanfeng Wang, Jian-Qing Shi+1 more|arXiv (Cornell University)|Nov 11, 2015

Advanced Statistical Methods and Models20 references3 citations

TL;DR

This paper proposes an Extended T-process Regression (eTPR) model that extends Gaussian Process Regression (GPR) by incorporating a heavy-tailed error structure through an extended t-process, enabling robust best linear unbiased prediction (BLUP) against outliers in both input and output spaces. The model retains closed-form predictive distributions and efficient computation via minor modifications to standard BLUP algorithms, outperforming GPR and LOESS in prediction accuracy under contamination.

ABSTRACT

Gaussian process regression (GPR) model has been widely used to fit data when the regression function is unknown and its nice properties have been well established. In this article, we introduce an extended t-process regression (eTPR) model, which gives a robust best linear unbiased predictor (BLUP). Owing to its succinct construction, it inherits many attractive properties from the GPR model, such as having closed forms of marginal and predictive distributions to give an explicit form for robust BLUP procedures, and easy to cope with large dimensional covariates with an efficient implementation by slightly modifying existing BLUP procedures. Properties of the robust BLUP are studied. Simulation studies and real data applications show that the eTPR model gives a robust fit in the presence of outliers in both input and output spaces and has a good performance in prediction, compared with the GPR and locally weighted scatterplot smoothing (LOESS) methods.

Motivation & Objective

• To develop a robust regression model that maintains predictive accuracy in the presence of outliers in both input and output spaces.
• To extend Gaussian Process Regression (GPR) with a heavy-tailed distribution that preserves closed-form predictive distributions and computational efficiency.
• To provide a robust BLUP procedure that selectively shrinks extreme observations, especially in sparsely sampled regions.
• To ensure the model remains computationally feasible for high-dimensional covariates by leveraging existing BLUP algorithm frameworks.

Proposed method

• The eTPR model uses an extended t-process (ETP) as a prior over the latent function, constructed via a scale mixture of Gaussians to induce heavy-tailed marginal distributions.
• The model assumes a hierarchical structure where the observation variance is stochastically weighted by a gamma-distributed precision parameter, enabling robustness to heavy-tailed errors.
• It derives closed-form expressions for the marginal and predictive distributions by integrating out latent variables, ensuring analytical tractability.
• The robust BLUP is computed via a modified version of the standard GPR BLUP algorithm, requiring only minor adjustments to existing implementations.
• The method employs selective shrinkage: observations with high leverage or large residuals are downweighted based on their influence on the posterior mean.
• The inference procedure uses empirical Bayes to estimate hyperparameters, including the degrees of freedom and scale parameters, via maximum marginal likelihood.

Experimental results

Research questions

• RQ1 Can a t-process-based regression model maintain closed-form predictive distributions while offering robustness to outliers in both input and output spaces?
• RQ2 How does the eTPR model compare to GPR and LOESS in terms of predictive performance under contamination?
• RQ3 Does the eTPR model exhibit selective shrinkage that reduces the influence of outliers without compromising fit in dense regions?
• RQ4 To what extent does the eTPR model remain computationally efficient for high-dimensional covariates compared to standard GPR?
• RQ5 Is the robust BLUP procedure under eTPR information-consistent and asymptotically optimal under model misspecification?

Key findings

• The eTPR model achieves robustness to outliers in both input and output spaces by leveraging the heavy-tailed nature of the extended t-process, outperforming GPR and LOESS in prediction accuracy under contamination.
• The model provides closed-form expressions for marginal and predictive distributions, enabling efficient computation without requiring MCMC or variational inference.
• Simulation results show that eTPR exhibits selective shrinkage, significantly reducing the influence of outliers at sparse input locations while preserving fit in dense regions.
• In real data applications, eTPR consistently delivers more stable and accurate predictions than GPR and LOESS, especially when data contain heavy-tailed noise or leverage points.
• The robust BLUP procedure under eTPR is information-consistent: as sample size increases, the predictive distribution converges to the true underlying function even under model misspecification.
• Theoretical analysis confirms that the eTPR model maintains asymptotic optimality in prediction under mild regularity conditions, with the divergence between true and estimated predictive distributions diminishing at rate o(n) in expectation.