[论文解读] Minimizing Negative Transfer of Knowledge in Multivariate Gaussian Processes: A Scalable and Regularized Approach

Raed Al Kontar, Garvesh Raskutti+1 more|arXiv (Cornell University)|Jan 31, 2019

Gaussian Processes and Bayesian Inference参考文献 53被引用 3

一句话总结

该论文提出了一种可扩展的、正则化的成对建模方法，用于多变量高斯过程（MGPs），通过卷积过程（CPs）实现，通过惩罚不相关输出之间的共享潜在函数来减轻负迁移。通过将完整的MGP分解为成对的双变量GPs，并应用组套索正则化，该方法在高维输出设置下确保了计算效率和预测精度的提升。

ABSTRACT

Recently there has been an increasing interest in the multivariate Gaussian process (MGP) which extends the Gaussian process (GP) to deal with multiple outputs. One approach to construct the MGP and account for non-trivial commonalities amongst outputs employs a convolution process (CP). The CP is based on the idea of sharing latent functions across several convolutions. Despite the elegance of the CP construction, it provides new challenges that need yet to be tackled. First, even with a moderate number of outputs, model building is extremely prohibitive due to the huge increase in computational demands and number of parameters to be estimated. Second, the negative transfer of knowledge may occur when some outputs do not share commonalities. In this paper we address these issues. We propose a regularized pairwise modeling approach for the MGP established using CP. The key feature of our approach is to distribute the estimation of the full multivariate model into a group of bivariate GPs which are individually built. Interestingly pairwise modeling turns out to possess unique characteristics, which allows us to tackle the challenge of negative transfer through penalizing the latent function that facilitates information sharing in each bivariate model. Predictions are then made through combining predictions from the bivariate models within a Bayesian framework. The proposed method has excellent scalability when the number of outputs is large and minimizes the negative transfer of knowledge between uncorrelated outputs. Statistical guarantees for the proposed method are studied and its advantageous features are demonstrated through numerical studies.

研究动机与目标

• 解决输出数量较多时多变量高斯过程（MGPs）的计算不可行性问题。
• 减轻输出不相关或弱相关时知识的负迁移问题。
• 开发一种可扩展的估计框架，同时保持统计精度和不确定性量化。
• 通过输出的成对建模实现MGPs的分布式估计。
• 在正则化条件下提供一致性与选择准确性的理论保证。

提出的方法

• 该方法将完整的多变量GP分解为一组双变量GP，每个双变量GP独立建模一对输出。
• 每个双变量GP通过一种正则化似然函数进行估计，该函数通过对卷积过程参数应用组套索来惩罚共享潜在函数。
• 正则化项旨在压缩促进不相关输出之间信息共享的潜在函数的影响。
• 预测结果在贝叶斯框架内通过所有成对模型的预测分布进行组合。
• 该方法利用泰勒展开和渐近分析，建立了估计量的理论一致性。
• 通过避免对整个输出集进行完整协方差矩阵求逆，确保了计算可扩展性。

实验结果

研究问题

• RQ1 一种可扩展且正则化的成对方法能否减轻多输出高斯过程中的计算负担？
• RQ2 当输出不相关时，如何最小化知识的负迁移？
• RQ3 带有正则化的成对建模是否能保持统计一致性和预测精度？
• RQ4 该方法在高维输出空间中能否实现模型选择的一致性？
• RQ5 在样本量和输出维度不断增加的情况下，正则化估计量的理论行为如何？

主要发现

• 所提出的方法通过将完整MGP分解为成对双变量模型，实现了计算可扩展性，避免了完整多变量估计的高昂代价。
• 通过组套索正则化有效抑制了不相关输出之间的信息共享，最小化了负迁移。
• 理论分析表明，随着样本量增加，估计量具有一致性，且无关潜在函数的选择概率趋近于1。
• 即使在输出维度较大的情况下，该方法仍能保持与完整MGP模型相当的不确定性量化和预测性能。
• 数值研究结果表明，在混合相关与不相关输出的情境下，该方法在性能上优于标准MGP和可分协方差方法。
• 理论结果证实，在满足正则性条件时，估计量以趋近于1的概率选择出真实的潜在结构。