[Paper Review] Spatio-Temporal Variational Gaussian Processes

Oliver Hamelijnck, William J. Wilkinson+3 more|Warwick Research Archive Portal (University of Warwick)|Nov 2, 2021

Gaussian Processes and Bayesian Inference49 references4 citations

TL;DR

This paper introduces Spatio-Temporal Variational Gaussian Processes (ST-VGP), a scalable method for large-scale spatio-temporal modeling that combines natural gradient variational inference with spatio-temporal filtering and smoothing. It achieves linear time complexity in the number of time steps and logarithmic span complexity via parallel filtering, while maintaining exact equivalence to standard variational GP under separable kernels, enabling efficient, accurate inference on massive spatio-temporal datasets with improved predictive performance through increased inducing points.

ABSTRACT

We introduce a scalable approach to Gaussian process inference that combines spatio-temporal filtering with natural gradient variational inference, resulting in a non-conjugate GP method for multivariate data that scales linearly with respect to time. Our natural gradient approach enables application of parallel filtering and smoothing, further reducing the temporal span complexity to be logarithmic in the number of time steps. We derive a sparse approximation that constructs a state-space model over a reduced set of spatial inducing points, and show that for separable Markov kernels the full and sparse cases exactly recover the standard variational GP, whilst exhibiting favourable computational properties. To further improve the spatial scaling we propose a mean-field assumption of independence between spatial locations which, when coupled with sparsity and parallelisation, leads to an efficient and accurate method for large spatio-temporal problems.

Motivation & Objective

• To address the cubic computational scaling of standard Gaussian processes in spatio-temporal settings.
• To enable scalable inference for multivariate, non-conjugate likelihoods in large-scale spatio-temporal data.
• To achieve linear time complexity in the number of time steps while preserving the accuracy of standard variational GP.
• To improve spatial scalability through mean-field approximation and sparsity.
• To demonstrate that spatio-temporal filtering with natural gradients enables efficient, accurate inference comparable to state-of-the-art variational GP.

Proposed method

• Employs natural gradient variational inference using sequential filtering and smoothing to achieve linear time complexity in the number of time steps.
• Derives a sparse approximation by constructing a state-space model over a reduced set of spatial inducing points.
• Utilizes parallel filtering and smoothing to reduce temporal span complexity to logarithmic in the number of time steps.
• Applies a mean-field assumption of independence across spatial locations to further improve spatial scalability.
• Reformulates the model to enable efficient computation under sparsity and parallelization, especially for separable Markov kernels.
• Uses a variational lower bound with natural gradients to optimize the approximate posterior, ensuring equivalence to standard SVGP under separable kernels.

Experimental results

Research questions

• RQ1 Can natural gradient variational inference be effectively combined with spatio-temporal filtering to achieve linear time complexity in the temporal dimension?
• RQ2 Does the proposed method maintain exact equivalence to standard variational GP under separable Markov kernels while improving scalability?
• RQ3 Can parallel filtering and smoothing reduce temporal span complexity from cubic to logarithmic in large spatio-temporal models?
• RQ4 How does the mean-field approximation across spatial locations affect predictive accuracy and computational efficiency?
• RQ5 Can the method scale to large datasets with significantly more inducing points than standard approaches, improving predictive performance?

Key findings

• ST-VGP achieves linear time complexity in the number of time steps, reducing computational cost from cubic to linear scaling.
• The method reduces temporal span complexity to logarithmic via parallel filtering and smoothing, enabling efficient inference on long time series.
• For separable Markov kernels, the full and sparse ST-VGP models exactly recover the standard variational GP, ensuring consistency with established methods.
• The mean-field approximation across spatial locations, combined with sparsity and parallelization, significantly improves spatial scalability without sacrificing accuracy.
• On synthetic data, ST-VGP achieves a test RMSE of 0.45 ± 0.11 on the largest dataset (dataset 5), outperforming SVGP and MF-ST-VGP, which show higher error and instability.
• On real-world datasets such as tree counts and air quality, ST-VGP achieves lower test negative log predictive likelihood (NLPD) with high-resolution binning, demonstrating improved predictive accuracy.