[Paper Review] Minimizing Negative Transfer of Knowledge in Multivariate Gaussian Processes: A Scalable and Regularized Approach

Raed Al Kontar, Garvesh Raskutti+1 more|arXiv (Cornell University)|Jan 31, 2019

Gaussian Processes and Bayesian InferenceComputer Science53 references3 citations

TL;DR

This paper proposes a scalable, regularized pairwise modeling approach for multivariate Gaussian processes (MGPs) using convolution processes (CPs), which mitigates negative transfer by penalizing shared latent functions between uncorrelated outputs. By decomposing the full MGP into bivariate GPs and applying group lasso regularization, the method ensures computational efficiency and improved prediction accuracy in high-dimensional output settings.

ABSTRACT

Recently there has been an increasing interest in the multivariate Gaussian process (MGP) which extends the Gaussian process (GP) to deal with multiple outputs. One approach to construct the MGP and account for non-trivial commonalities amongst outputs employs a convolution process (CP). The CP is based on the idea of sharing latent functions across several convolutions. Despite the elegance of the CP construction, it provides new challenges that need yet to be tackled. First, even with a moderate number of outputs, model building is extremely prohibitive due to the huge increase in computational demands and number of parameters to be estimated. Second, the negative transfer of knowledge may occur when some outputs do not share commonalities. In this paper we address these issues. We propose a regularized pairwise modeling approach for the MGP established using CP. The key feature of our approach is to distribute the estimation of the full multivariate model into a group of bivariate GPs which are individually built. Interestingly pairwise modeling turns out to possess unique characteristics, which allows us to tackle the challenge of negative transfer through penalizing the latent function that facilitates information sharing in each bivariate model. Predictions are then made through combining predictions from the bivariate models within a Bayesian framework. The proposed method has excellent scalability when the number of outputs is large and minimizes the negative transfer of knowledge between uncorrelated outputs. Statistical guarantees for the proposed method are studied and its advantageous features are demonstrated through numerical studies.

Motivation & Objective

• To address the computational infeasibility of multivariate Gaussian processes (MGPs) with many outputs.
• To mitigate negative transfer of knowledge when outputs are uncorrelated or weakly correlated.
• To develop a scalable estimation framework that maintains statistical accuracy and uncertainty quantification.
• To enable distributed estimation of MGPs through pairwise modeling of outputs.
• To provide theoretical guarantees on consistency and selection accuracy under regularization.

Proposed method

• The method decomposes the full multivariate GP into a set of bivariate GPs, each modeling a pair of outputs independently.
• Each bivariate GP is estimated using a regularized likelihood that penalizes shared latent functions via group lasso on the convolution process parameters.
• The regularization term is designed to shrink the influence of latent functions that facilitate information sharing between uncorrelated outputs.
• Predictions are combined within a Bayesian framework using predictive distributions from all pairwise models.
• The approach leverages a Taylor expansion and asymptotic analysis to establish theoretical consistency of the estimator.
• The method ensures computational scalability by avoiding full covariance matrix inversion for the entire output set.

Experimental results

Research questions

• RQ1 Can a scalable and regularized approach reduce computational burden in multivariate Gaussian processes with many outputs?
• RQ2 How can negative transfer of knowledge be minimized when outputs are uncorrelated?
• RQ3 Does pairwise modeling with regularization maintain statistical consistency and prediction accuracy?
• RQ4 Can the method achieve model selection consistency in high-dimensional output spaces?
• RQ5 What is the theoretical behavior of the regularized estimator under increasing sample size and output dimension?

Key findings

• The proposed method achieves computational scalability by decomposing the full MGP into pairwise bivariate models, avoiding the prohibitive cost of full multivariate estimation.
• Regularization via group lasso effectively suppresses information sharing between uncorrelated outputs, minimizing negative transfer.
• Theoretical analysis shows that the estimator is consistent and selection of irrelevant latent functions is achieved with high probability as sample size increases.
• The method maintains uncertainty quantification and predictive performance comparable to full MGP models, even with large output dimensions.
• Numerical studies demonstrate superior performance over standard MGP and separable covariance approaches in scenarios with mixed correlated and uncorrelated outputs.
• Theoretical results confirm that the estimator selects the true underlying structure with probability approaching one under regularity conditions.