[Paper Review] Distributed Gaussian Processes

Marc Peter Deisenroth, Jun Wei Ng|arXiv (Cornell University)|Feb 10, 2015

Gaussian Processes and Bayesian InferenceComputer Science29 references139 citations

TL;DR

This paper proposes the robust Bayesian Committee Machine (rBCM), a distributed Gaussian process model that scales to arbitrarily large datasets by splitting data across independent computational units, training local GP experts, and recursively combining predictions via closed-form product-of-experts inference. The method achieves state-of-the-art performance on large-scale regression tasks, including a 6M-flight dataset, with sub-30-minute training on a laptop and O(10^7) data scalability without inducing points or variational parameters.

ABSTRACT

Copyright © 2015 by the author(s).To scale Gaussian processes (GPs) to large data sets we introduce the robust Bayesian Committee Machine (rBCM), a practical and scalable product-of-experts model for large-scale distributed GP regression. Unlike state-of-the-art sparse GP approximations, the rBCM is conceptually simple and does not rely on inducing or variational parameters. The key idea is to recursively distribute computations to independent computational units and, subsequently, re-combine them to form an overall result. Efficient closed-form inference allows for straightforward parallelisation and distributed computations with a small memory footprint. The rBCM is independent of the computational graph and can be used on heterogeneous computing infrastructures, ranging from laptops to clusters. With sufficient computing resources our distributed GP model can handle arbitrarily large data sets.

Motivation & Objective

• To address the O(N³) computational and O(N²) memory bottlenecks of standard Gaussian processes on large datasets.
• To develop a scalable, distributed GP framework that avoids sparse approximations relying on inducing or variational parameters.
• To enable efficient, parallelized GP inference on heterogeneous hardware, from laptops to clusters, without sacrificing predictive accuracy.
• To provide a consistent, analytically tractable alternative to mixture-of-experts and PoE-GP models that avoids overconfidence or over-conservatism.

Proposed method

• The rBCM partitions the full training dataset into disjoint subsets and trains independent GP experts on each subset using shared hyper-parameters.
• It employs a recursive tree-structured aggregation of predictions from local experts, combining them via a product-of-experts (PoE) formulation to form a global posterior.
• All inference is performed in closed form using analytical solutions for mean and variance, avoiding MCMC or sampling-based approximations.
• The model uses a hierarchical, distributed computation graph that is independent of hardware topology, enabling deployment on diverse infrastructures.
• The overall predictive distribution is derived by multiplying the predictive densities of individual experts, weighted naturally by their reliability.
• Hyperparameters are optimized via gradient ascent on the log-marginal likelihood, with the full gradient computed as an average of expert gradients, promoting regularization.

Experimental results

Research questions

• RQ1 Can a distributed GP model scale to O(10^7) data points without relying on inducing points or variational parameters?
• RQ2 How does a product-of-experts GP model compare to sparse GP methods in terms of predictive accuracy and computational efficiency on large, non-stationary datasets?
• RQ3 Can recursive, closed-form inference in a distributed PoE framework maintain uncertainty calibration while enabling parallelization across heterogeneous systems?
• RQ4 Does the rBCM avoid the overconfidence of standard PoE-GP and over-conservatism of generalized PoE-GP models in real-world regression tasks?
• RQ5 Can the rBCM achieve high performance with minimal memory footprint and fast convergence on commodity hardware?

Key findings

• The rBCM achieved an RMSE of 27.1 and NLPD of 9.1 on a 700K-flight data subset, outperforming SVIGP and Dist-VGP in both metrics.
• On a 5M-flight data set, the rBCM achieved RMSE 35.5 and NLPD 8.8, demonstrating robustness and scalability to large-scale, non-stationary data.
• Training a million-point GP model on a laptop completed in under 30 minutes, highlighting low memory and computational overhead.
• The rBCM showed consistent performance across all data sizes (700K, 2M, 5M), with standard errors below 0.3, indicating stable optimization.
• The model significantly outperformed the standard BCM and PoE models, which suffered from overconfidence (PoE) and weak-expert issues (BCM).
• The rBCM’s NLPD values remained stable across increasing data sizes, suggesting reliable uncertainty estimates even as data complexity grew.