[Paper Review] When Gaussian Process Meets Big Data: A Review of Scalable GPs

Haitao Liu, Yew-Soon Ong+2 more|arXiv (Cornell University)|Jul 3, 2018

Gaussian Processes and Bayesian InferenceComputer Science248 references98 citations

TL;DR

A comprehensive survey of scalable Gaussian Process methods for large-scale regression, categorizing global and local approximations and their trade-offs in accuracy and efficiency.

ABSTRACT

The vast quantity of information brought by big data as well as the evolving computer hardware encourages success stories in the machine learning community. In the meanwhile, it poses challenges for the Gaussian process (GP) regression, a well-known non-parametric and interpretable Bayesian model, which suffers from cubic complexity to data size. To improve the scalability while retaining desirable prediction quality, a variety of scalable GPs have been presented. But they have not yet been comprehensively reviewed and analyzed in order to be well understood by both academia and industry. The review of scalable GPs in the GP community is timely and important due to the explosion of data size. To this end, this paper is devoted to the review on state-of-the-art scalable GPs involving two main categories: global approximations which distillate the entire data and local approximations which divide the data for subspace learning. Particularly, for global approximations, we mainly focus on sparse approximations comprising prior approximations which modify the prior but perform exact inference, posterior approximations which retain exact prior but perform approximate inference, and structured sparse approximations which exploit specific structures in kernel matrix; for local approximations, we highlight the mixture/product of experts that conducts model averaging from multiple local experts to boost predictions. To present a complete review, recent advances for improving the scalability and capability of scalable GPs are reviewed. Finally, the extensions and open issues regarding the implementation of scalable GPs in various scenarios are reviewed and discussed to inspire novel ideas for future research avenues.

Motivation & Objective

• Assess the challenges of applying Gaussian Process regression to big data due to cubic complexity.
• Categorize scalable GPs into global and local approximations and analyze their trade-offs in accuracy and scalability.
• Review key global approaches (subset-of-data, sparse kernels, sparse approximations) and local approaches (mixture/product of experts).
• Summarize advances in improving scalability and model capability, and discuss extensions and open issues.

Proposed method

• Classify scalable GPs into global approximations and local approximations.
• Detail global approaches: subset-of-data, sparse kernels, and sparse approximations with inducing points and Nyström methods.
• Explain posterior approximations such as variational free energy (VFE) and stochastic variational GP (SVGP) for scalable inference.
• Describe inter-domain, hierarchical, and hybrid global-local strategies to improve capability.
• Discuss stochastic optimization and inducing-point optimization to scale to million- or billion-sized data.

Experimental results

Research questions

• RQ1 What are the main scalable GP paradigms for large-scale regression?
• RQ2 How do global and local approximations compare in terms of scalability and predictive capability?
• RQ3 What advances (e.g., variational methods, inducing points, and stochastic optimization) enable handling very large datasets with GPs?
• RQ4 What are the open issues and future directions for scalable GPs in practice?

Key findings

• Global approximations enable substantial scalability by reducing kernel matrix size or structure while aiming to preserve global patterns.
• Local approximations capture local patterns and non-stationarities but may miss global structure, motivating hybrids.
• Variational approaches (VFE, SVGP) provide principled, scalable inference and can recover the full GP with enough inducing points.
• Stochastic optimization and inducing-point methods enable training on data sizes from millions to billions.
• Structured inducing sets and inter-domain strategies yield improved scalability and model capability, with ongoing trade-offs in uncertainty estimation.