[Paper Review] Hierarchical Mixture-of-Experts Model for Large-Scale Gaussian Process Regression

Jun Wei Ng, Marc Peter Deisenroth|arXiv (Cornell University)|Dec 9, 2014

Gaussian Processes and Bayesian InferenceComputer Science23 references25 citations

TL;DR

This paper proposes a hierarchical mixture-of-experts Gaussian process model that scales full GP regression to massive data sets by distributing computations across independent computational units. Using closed-form, analytically recombined predictions from local GP experts in a tree-structured architecture, the method achieves O(N³) training complexity per leaf but enables massive parallelization, allowing training on over 10⁷ data points with low memory usage and fast convergence—demonstrated on a 1.7×10⁷-point data set and a million-point data set in under 30 minutes on a laptop.

ABSTRACT

We propose a practical and scalable Gaussian process model for large-scale nonlinear probabilistic regression. Our mixture-of-experts model is conceptually simple and hierarchically recombines computations for an overall approximation of a full Gaussian process. Closed-form and distributed computations allow for efficient and massive parallelisation while keeping the memory consumption small. Given sufficient computing resources, our model can handle arbitrarily large data sets, without explicit sparse approximations. We provide strong experimental evidence that our model can be applied to large data sets of sizes far beyond millions. Hence, our model has the potential to lay the foundation for general large-scale Gaussian process research.

Motivation & Objective

• Address the computational and memory limitations of standard Gaussian process regression on large data sets (N > 10,000).
• Develop a scalable, practical alternative to sparse GP approximations that avoids explicit inducing point or low-rank approximations.
• Enable full GP inference on arbitrarily large data sets by leveraging massive parallelization across independent computational units.
• Maintain the predictive accuracy and probabilistic coherence of a full GP while drastically reducing per-node computational load.
• Provide a framework that supports full GP training with minimal hyperparameter tuning and no sampling-based inference.

Proposed method

• Distributes training data across c local GP experts (leaf nodes), each performing full GP computations on a subset of the data.
• Uses a hierarchical tree structure to recursively recombine local predictions into global mean and variance estimates via closed-form analytical operations.
• Employs shared hyper-parameters (σf, li, σϵ) across all local experts to maintain consistency with a full GP and reduce overfitting.
• Performs recursive recombination of predictions using weighted sums of local means and variances, with weights derived from the gating mechanism at each level.
• Enables distributed training via Map-Reduce-style architecture, where each node computes locally and results are combined analytically without sampling.
• Caches the inverse of the kernel matrix (K + σ²ϵI)⁻¹ at each leaf to accelerate prediction, reducing per-prediction cost to O(N) for mean and O(N²) for variance.

Experimental results

Research questions

• RQ1 Can a hierarchical mixture-of-experts model scale full Gaussian process regression to data sets of 10⁷ points or more without sparse approximations?
• RQ2 How does the predictive performance of the proposed model compare to state-of-the-art sparse GP methods in terms of accuracy and convergence speed?
• RQ3 To what extent can the model achieve computational and memory efficiency through massive parallelization while preserving the statistical properties of a full GP?
• RQ4 Does the use of shared hyper-parameters across local experts prevent overfitting and improve optimization stability compared to models with per-expert parameters?
• RQ5 Can the model be trained efficiently on standard hardware (e.g., a laptop) for data sets of 1 million points or more?

Key findings

• The model trained a full GP on 1.7×10⁷ data points (kin40K) with a likelihood ratio of 0.815 compared to the ground truth GP, demonstrating scalability beyond typical sparse GP limits.
• Training a million-point data set took less than 30 minutes on a laptop, with convergence in just a few tens of BFGS iterations—significantly faster than sparse GP methods requiring hundreds or thousands of iterations.
• On the airline delays data set (700,000 training points), the HGP achieved a lower RMSE than both sparse variational GP (SVGP) and distributed sparse variational GP (Dist SVGP) methods, indicating superior predictive performance.
• The model achieved a likelihood ratio of 0.992 on the kin40K data set with only 4 leaves (5,000 points per leaf), showing strong approximation fidelity even with minimal hierarchy.
• The hierarchical recombination process achieved O(N) mean prediction and O(N²) variance prediction per test point, with memory usage kept low by avoiding full matrix storage.
• The model’s performance remained stable and accurate across multiple data sets and hierarchical depths, with likelihood ratios decreasing gradually as the number of leaves increased, indicating consistent approximation quality.