[Paper Review] Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models

Yarin Gal, Mark van der Wilk+1 more|arXiv (Cornell University)|Feb 6, 2014

Gaussian Processes and Bayesian Inference16 references84 citations

TL;DR

This paper proposes a distributed variational inference algorithm for sparse Gaussian process regression and latent variable models, leveraging a reparameterization that decouples data given inducing points to enable efficient Map-Reduce parallelization. The method scales to 2 million data points, improves GP performance over baselines on flight and MNIST data, and maintains load balance across nodes with minimal communication overhead.

ABSTRACT

Gaussian processes (GPs) are a powerful tool for probabilistic inference over functions. They have been applied to both regression and non-linear dimensionality reduction, and offer desirable properties such as uncertainty estimates, robustness to over-fitting, and principled ways for tuning hyper-parameters. However the scalability of these models to big datasets remains an active topic of research. We introduce a novel re-parametrisation of variational inference for sparse GP regression and latent variable models that allows for an efficient distributed algorithm. This is done by exploiting the decoupling of the data given the inducing points to re-formulate the evidence lower bound in a Map-Reduce setting. We show that the inference scales well with data and computational resources, while preserving a balanced distribution of the load among the nodes. We further demonstrate the utility in scaling Gaussian processes to big data. We show that GP performance improves with increasing amounts of data in regression (on flight data with 2 million records) and latent variable modelling (on MNIST). The results show that GPs perform better than many common models often used for big data.

Motivation & Objective

• To address the scalability limitations of sparse Gaussian process models on big data.
• To enable efficient, load-balanced distributed inference for GP regression and latent variable models.
• To demonstrate that GP performance improves with increasing data volume using scalable inference.
• To develop a distributed algorithm that maintains the tight variational lower bound of Titsias (2009) while enabling parallelization.
• To show practical utility of large-scale GPs on real-world datasets like flight data and MNIST.

Proposed method

• Reformulates the evidence lower bound (ELBO) using a reparameterization that decouples data given inducing points, enabling independent node-wise updates.
• Applies the Map-Reduce paradigm to distribute computation across nodes, with only constant-time communication between nodes.
• Uses the analytic optimal form of the variational distribution over inducing targets, preserving the tightness of the ELBO from Titsias (2009).
• Enables parallel optimization of latent variable embeddings in the GPLVM by decoupling data dependencies.
• Implements the algorithm in Python using the Map-Reduce framework, with open-source availability.
• Employs LBFGS and SCG optimizers for hyperparameter and inducing point optimization, with careful initialization strategies.

Experimental results

Research questions

• RQ1 Can variational inference in sparse GP regression and GPLVMs be efficiently distributed while preserving the tightness of the ELBO?
• RQ2 How does the proposed distributed algorithm scale with increasing data size and computational resources?
• RQ3 Does training Gaussian processes on larger datasets (e.g., 2M points) lead to improved predictive performance compared to standard models?
• RQ4 Can the algorithm maintain balanced load distribution and low communication overhead in a distributed setting?
• RQ5 How does the performance of GPs compare to common big-data models like random forests and linear models when trained on large datasets?

Key findings

• The proposed distributed inference scaled to 2 million flight records, achieving an RMSE of 35.31, outperforming linear regression (37.65), ridge regression (37.65), random forest (37.33), and even SVI with 200 inducing points.
• On the full MNIST dataset (60K points), the GPLVM reduced classification error from 8.98% to 5.95% when training on all data versus a subset, demonstrating improved performance with more data.
• Training on 2M points with 64 cores took approximately 13.8 minutes per iteration, showing favorable scaling with computational resources.
• The LBFGS optimizer outperformed SCG in convergence stability, especially on large datasets, where SCG converged to poor optima due to sharpening likelihood modes.
• The algorithm achieved balanced load distribution across nodes and required only constant-time communication between nodes, enabling efficient parallelization.
• The open-source implementation demonstrated robustness to node failure through random dropout experiments in the supplementary material.