[论文解读] When Gaussian Process Meets Big Data: A Review of Scalable GPs

Haitao Liu, Yew-Soon Ong+2 more|arXiv (Cornell University)|Jul 3, 2018

Gaussian Processes and Bayesian Inference参考文献 248被引用 98

一句话总结

关于大规模回归的可扩展高斯过程方法的综合综述，分类全局近似和局部近似及它们在精度与效率之间的权衡。

ABSTRACT

The vast quantity of information brought by big data as well as the evolving computer hardware encourages success stories in the machine learning community. In the meanwhile, it poses challenges for the Gaussian process (GP) regression, a well-known non-parametric and interpretable Bayesian model, which suffers from cubic complexity to data size. To improve the scalability while retaining desirable prediction quality, a variety of scalable GPs have been presented. But they have not yet been comprehensively reviewed and analyzed in order to be well understood by both academia and industry. The review of scalable GPs in the GP community is timely and important due to the explosion of data size. To this end, this paper is devoted to the review on state-of-the-art scalable GPs involving two main categories: global approximations which distillate the entire data and local approximations which divide the data for subspace learning. Particularly, for global approximations, we mainly focus on sparse approximations comprising prior approximations which modify the prior but perform exact inference, posterior approximations which retain exact prior but perform approximate inference, and structured sparse approximations which exploit specific structures in kernel matrix; for local approximations, we highlight the mixture/product of experts that conducts model averaging from multiple local experts to boost predictions. To present a complete review, recent advances for improving the scalability and capability of scalable GPs are reviewed. Finally, the extensions and open issues regarding the implementation of scalable GPs in various scenarios are reviewed and discussed to inspire novel ideas for future research avenues.

研究动机与目标

• 评估将高斯过程回归应用于大数据时由于三次复杂度带来的挑战。
• 将可扩展的GP分为全局近似和局部近似，并分析它们在精度与可扩展性方面的权衡。
• 回顾关键的全局方法（数据子集、稀疏核、以及带有诱导点和Nyström方法的稀疏近似）和局部方法（专家混合/专家乘积）。
• 总结在提升可扩展性和模型能力方面的进展，并讨论扩展与未解问题。

提出的方法

• 将可扩展GP分为全局近似和局部近似。
• 详细描述全局方法：数据子集、稀疏核，以及带有诱导点和Nyström方法的稀疏近似。
• 解释后验近似，如变分自由能（VFE）和随机变分GP（SVGP），用于可扩展推断。
• 描述跨域、层次和全局−局部混合策略以提升能力。
• 讨论随机优化和诱导点优化以扩展到百万级或十亿级数据。

实验结果

研究问题

• RQ1 针对大规模回归，主要的可扩展GP范式是什么？
• RQ2 在可扩展性和预测能力方面，全局近似和局部近似的比较如何？
• RQ3 哪些进展（如变分方法、诱导点和随机优化）使GP能够处理极大数据集？
• RQ4 在实际中可扩展GP的未解问题和未来方向是什么？

主要发现

• 全局近似通过减小核矩阵大小或结构来实现显著的可扩展性，同时力图保留全局模式。
• 局部近似捕捉局部模式和非平稳性，但可能错过全局结构，促成混合方法。
• 变分方法（VFE、SVGP）提供有原则的、可扩展的推断，并且用足够的诱导点可以恢复完整GP。
• 随机优化和诱导点方法使训练能够处理从数百万到十亿级的数据规模。
• 结构化诱导集和跨域策略在提升可扩展性和模型能力方面有改进，同时在不确定性估计方面仍有权衡。