[Paper Review] Sparse Gaussian Process Variational Autoencoders

Matthew Ashman, Jonathan So+4 more|arXiv (Cornell University)|Oct 20, 2020

Gaussian Processes and Bayesian Inference4 citations

TL;DR

This paper introduces the Sparse Gaussian Process Variational Autoencoder (SGP-VAE), a deep generative model that integrates sparse Gaussian processes with variational autoencoders using inference networks to enable efficient, amortized inference over multi-output sparse GPs. It supports missing data handling via partial inference networks and outperforms existing methods in modeling large, multi-dimensional spatio-temporal datasets with complex dependencies.

ABSTRACT

Large, multi-dimensional spatio-temporal datasets are omnipresent in modern science and engineering. An effective framework for handling such data are Gaussian process deep generative models (GP-DGMs), which employ GP priors over the latent variables of DGMs. Existing approaches for performing inference in GP-DGMs do not support sparse GP approximations based on inducing points, which are essential for the computational efficiency of GPs, nor do they handle missing data -- a natural occurrence in many spatio-temporal datasets -- in a principled manner. We address these shortcomings with the development of the sparse Gaussian process variational autoencoder (SGP-VAE), characterised by the use of partial inference networks for parameterising sparse GP approximations. Leveraging the benefits of amortised variational inference, the SGP-VAE enables inference in multi-output sparse GPs on previously unobserved data with no additional training. The SGP-VAE is evaluated in a variety of experiments where it outperforms alternative approaches including multi-output GPs and structured VAEs.

Motivation & Objective

• To address the lack of support for sparse GP approximations and principled missing data handling in existing GP-DGMs.
• To enable efficient, amortized inference over multi-output sparse GPs on previously unobserved data.
• To develop a unified framework that combines the strengths of deep generative models and Gaussian processes for large-scale spatio-temporal data.
• To support flexible likelihoods and structured dependencies in multi-dimensional outputs while maintaining computational scalability.

Proposed method

• Proposes the SGP-VAE, which uses inference networks to parameterize multi-output sparse GP approximations via inducing points.
• Employs partial inference networks—PointNet, IndexNet, and FactorNet—to handle missing data by conditioning on observed outputs only.
• Leverages amortized variational inference to enable fast, generalization-capable inference on unseen data without retraining.
• Uses a factorized likelihood in the decoder to model complex, multi-dimensional outputs with flexible likelihoods.
• Applies k-means clustering to initialize shared inducing point locations across latent dimensions and tasks.
• Employs deep neural networks with ReLU or tanh activations to parameterize the encoder, decoder, and inference networks.

Experimental results

Research questions

• RQ1 Can sparse GP approximations be effectively integrated into variational autoencoders to scale to large, multi-dimensional spatio-temporal datasets?
• RQ2 How can missing data be handled in a principled and scalable way within deep generative models with GP priors?
• RQ3 Does the proposed SGP-VAE achieve better predictive performance than multi-output GPs and structured VAEs on real-world spatio-temporal data?
• RQ4 Can amortized inference in the SGP-VAE generalize to previously unobserved data without additional training?

Key findings

• The SGP-VAE outperforms baseline multi-output GPs and structured VAEs in terms of predictive performance on the bouncing ball and weather station datasets.
• On the weather station dataset, the SGP-VAE achieved a lower root mean squared error (RMSE) and negative log-likelihood (NLL) compared to other models, with 93.2% missing data in some variables.
• The model demonstrated strong generalization, enabling accurate inference on previously unobserved data through amortized inference.
• The partial inference networks (PointNet, IndexNet, FactorNet) effectively handled missing data, with FactorNet showing particular robustness in high-missingness scenarios.
• The SGP-VAE achieved state-of-the-art performance on the bouncing ball task, with near-exact recovery of ground truth trajectories using only a single latent dimension.