[Paper Review] Scalable Gaussian Process Variational Autoencoders

Jazbec, Metod, Matthew Ashman+4 more|arXiv (Cornell University)|Oct 26, 2020

Gaussian Processes and Bayesian Inference57 references16 citations

TL;DR

This paper proposes a scalable Gaussian Process Variational Autoencoder (GP-VAE) using a novel hybrid sparse inference approach based on inducing points, enabling efficient mini-batch training and end-to-end optimization. The method reduces computational complexity from O(N³) to O(bm² + m³), where b is batch size and m is number of inducing points, while maintaining high accuracy and supporting arbitrary kernel choices and data types without architectural modifications.

ABSTRACT

Conventional variational autoencoders fail in modeling correlations between data points due to their use of factorized priors. Amortized Gaussian process inference through GP-VAEs has led to significant improvements in this regard, but is still inhibited by the intrinsic complexity of exact GP inference. We improve the scalability of these methods through principled sparse inference approaches. We propose a new scalable GP-VAE model that outperforms existing approaches in terms of runtime and memory footprint, is easy to implement, and allows for joint end-to-end optimization of all components.

Motivation & Objective

• To address the scalability limitations of existing GP-VAE models that rely on exact GP inference and suffer from O(N³) complexity.
• To enable mini-batch training and amortized inference in GP-VAEs, which existing sparse GP methods fail to support due to memory or amortization constraints.
• To develop a generic, kernel-agnostic framework that supports joint end-to-end optimization of all model components, including GP hyperparameters and VAE parameters.
• To provide a theoretically grounded, easy-to-implement solution that maintains high performance while significantly reducing memory and runtime costs.

Proposed method

• Proposes a hybrid sparse GP approximation that combines variational inference over inducing points with amortized inference, enabling both mini-batching and end-to-end training.
• Adapts the stochastic variational inference framework from Hensman et al. (2013) to GP-VAEs by redefining the variational posterior over latent functions using inducing points.
• Derives a lower bound on the marginal likelihood that allows joint optimization of the VAE encoder, decoder, and GP prior, using a differentiable approximation of the GP marginal likelihood.
• Employs a structured variational posterior over the latent functions, parameterized by the VAE’s inference network, to enable amortized inference across all data points.
• Uses a reparameterization trick to enable backpropagation through the GP prior, allowing gradient-based optimization of all model components.
• Introduces a novel ELBO formulation that integrates the GP prior with the VAE objective, enabling joint optimization of the GP hyperparameters and VAE parameters.

Experimental results

Research questions

• RQ1 Can a sparse GP approximation be made compatible with both mini-batching and amortized inference in GP-VAE models?
• RQ2 Can the computational complexity of GP-VAEs be reduced from O(N³) to O(bm² + m³) without sacrificing model expressiveness or requiring kernel-specific design?
• RQ3 Is it possible to construct a generic, kernel-agnostic GP-VAE framework that supports joint end-to-end optimization of all components, including GP hyperparameters?
• RQ4 How does the proposed method compare in performance and efficiency to existing GP-VAE models that rely on exact inference or specialized kernels?

Key findings

• The proposed GP-VAE achieves a significant reduction in computational complexity from O(N³) to O(bm² + m³), enabling scalable training on large datasets.
• The model maintains high performance across multiple benchmarks, outperforming existing GP-VAE approaches in terms of both runtime and memory efficiency.
• The method supports joint end-to-end optimization of all components, including GP hyperparameters, unlike prior approaches that fix GP parameters or require separate optimization steps.
• Empirical results show that the model generalizes well even with larger numbers of inducing points, though bias in the approximation increases slightly in higher dimensions.
• The model is easy to implement and requires no special modifications to the training procedure, making it accessible for practitioners.