[论文解读] Generalized Robust Bayesian Committee Machine for Large-scale Gaussian Process Regression

Haitao Liu, Jianfei Cai+2 more|arXiv (Cornell University)|Jun 2, 2018

Gaussian Processes and Bayesian Inference被引用 42

一句话总结

GRBCM 为大规模高斯过程回归提供一致且高效的聚合方法，通过引入全局通信专家与增强预测，在一致性和可扩展性方面优于现有的 PoE/BCM 方法。

ABSTRACT

In order to scale standard Gaussian process (GP) regression to large-scale datasets, aggregation models employ factorized training process and then combine predictions from distributed experts. The state-of-the-art aggregation models, however, either provide inconsistent predictions or require time-consuming aggregation process. We first prove the inconsistency of typical aggregations using disjoint or random data partition, and then present a consistent yet efficient aggregation model for large-scale GP. The proposed model inherits the advantages of aggregations, e.g., closed-form inference and aggregation, parallelization and distributed computing. Furthermore, theoretical and empirical analyses reveal that the new aggregation model performs better due to the consistent predictions that converge to the true underlying function when the training size approaches infinity.

研究动机与目标

• Motivate the need for scalable GP regression on large datasets.
• Show inconsistencies of existing aggregation methods under disjoint or random partitions.
• Propose the Generalized Robust Bayesian Committee Machine (GRBCM) with a global communication expert.
• Prove consistency of GRBCM as data size grows and analyze its complexity.
• Demonstrate empirical performance advantages on toy and real datasets.

提出的方法

• Review and critique existing aggregation models (PoE, GPoE, BCM, RBCM, NPAE) and their inconsistencies.
• Introduce GRBCM that splits experts into a global communication expert plus enhanced experts.
• Formulate GRBCM prediction via Bayes rule with conditional independence and beta-weights.
• Define beta_i based on entropy differences to weight expert contributions.
• Provide theoretical proofs of consistency for GRBCM as n → ∞.
• Offer complexity analysis and discuss implementation in distributed computing environments.

实验结果

研究问题

• RQ1 Do common aggregation models yield consistent predictions when training size grows to infinity under various partitions?
• RQ2 Can a new aggregation model be designed to be both consistent and efficient for large-scale GP regression?
• RQ3 Does GRBCM provide improved predictive accuracy and calibrated uncertainty compared to existing aggregations?
• RQ4 What are the computational trade-offs of GRBCM in prediction and how does it scale with data size and number of experts?
• RQ5 Do empirical results on toy and real datasets support the theoretical consistency and performance claims?

主要发现

• GRBCM yields consistent predictions with increasing data, converging to the true underlying function and noise variance.
• GRBCM outperforms existing aggregations in accuracy (SMSE) and uncertainty quality (MSLL) on toy and real datasets.
• GRBCM’s prediction uncertainty remains controlled and generally lower than GPoE under random partitions.
• GRBCM maintains parallelizable, distributed computation with higher predictive accuracy at similar or modestly increased prediction time.
• NPAE remains effective but is significantly more time-consuming, making GRBCM a preferable scalable alternative.
• The approach demonstrates improved performance across toy, kin40k, and sarcos datasets.