[Paper Review] Scalable Variational Gaussian Process Classification

James Hensman, Alexander Matthews+1 more|arXiv (Cornell University)|Nov 7, 2014

Gaussian Processes and Bayesian Inference26 references346 citations

TL;DR

This paper proposes a scalable variational inference method for Gaussian process classification using a novel variational bound that directly approximates the marginal likelihood without pre-approximating the covariance matrix. By combining sparse inducing points with stochastic optimization, the method achieves state-of-the-art performance on benchmark datasets and scales to millions of data points, enabling accurate classification on large-scale problems like flight delay prediction.

ABSTRACT

Gaussian process classification is a popular method with a number of appealing properties. We show how to scale the model within a variational inducing point framework, outperforming the state of the art on benchmark datasets. Importantly, the variational formulation can be exploited to allow classification in problems with millions of data points, as we demonstrate in experiments.

Motivation & Objective

• To address the lack of scalable, variational inference methods for non-conjugate GP classification that can handle large datasets.
• To develop a variational bound that avoids pre-approximating the covariance matrix, improving approximation quality and optimization stability.
• To enable stochastic optimization of the variational objective, allowing training on datasets with millions of data points.
• To provide a unified framework compatible with complex GP models such as deep GPs and GP regression networks.
• To outperform existing methods like GFITC and IVM in both accuracy and scalability on benchmark and real-world datasets.

Proposed method

• Proposes a new variational bound, KLSP, that directly approximates the marginal likelihood without first approximating the full covariance matrix.
• Uses a sparse approximation with inducing inputs Z and inducing variables u, where q(f|u) is a Gaussian process conditioned on u.
• Employs a variational distribution q(u) = N(u|m, S) to approximate the joint posterior over inducing variables.
• Derives a lower bound on the log-marginal likelihood using Jensen's inequality, leading to a tractable objective function.
• Applies stochastic optimization via mini-batch updates using the ADADELTA algorithm, enabling training on large datasets like MNIST and flight data.
• Uses Gauss-Hermite quadrature to approximate the log-likelihood term in the variational objective for non-Gaussian likelihoods.

Experimental results

Research questions

• RQ1 Can a variational inference framework be designed for GP classification that avoids pre-approximating the full covariance matrix?
• RQ2 Can such a framework be optimized stochastically to scale to datasets with millions of data points?
• RQ3 Does the proposed method outperform existing sparse GP classification methods in terms of predictive accuracy and calibration?
• RQ4 Can the method be integrated into hierarchical GP models such as deep GPs or GP regression networks?
• RQ5 How well does the method generalize to real-world, high-dimensional, large-scale datasets like flight delay prediction?

Key findings

• The proposed KLSP method outperforms the state-of-the-art GFITC method on benchmark datasets, achieving lower negative log-probability and better calibration.
• On the MNIST dataset with 60,000 training points, the method achieved a test accuracy of 97.8% and a negative log-probability of 0.069 using stochastic optimization.
• On a real-world flight delay dataset of 5.9 million flights, the method significantly outperformed a linear classifier (37% error rate) with a negative log-probability of 0.58, indicating superior predictive performance.
• The optimized kernel parameters revealed that the Matern-3/2 kernel dominated, with time-of-day and time-of-year as the most relevant features, indicating high non-linearity in the data.
• The method demonstrated stable optimization behavior, unlike the mean-field sparse approach, which showed erratic performance due to poor calibration.
• The KLSP bound enables stochastic optimization, making GP classification feasible for big data applications previously intractable with standard GP methods.