[Paper Review] Variational zero-inflated Gaussian processes with sparse kernels

Pashupati Hegde, Markus Heinonen+1 more|arXiv (Cornell University)|Mar 13, 2018

Gaussian Processes and Bayesian Inference21 references3 citations

TL;DR

This paper proposes variational zero-inflated Gaussian processes (ZiGP) with sparse kernels that model excess zeros in datasets by learning a latent probit Gaussian process to sparsely zero out kernel rows and columns, enabling exact zero predictions. The method uses stochastic variational inference for scalable learning and introduces sparse Gaussian process networks (sGPRN) with variable-order mixtures, improving both prediction accuracy on zero-inflated data and interpretability of latent structures.

ABSTRACT

Zero-inflated datasets, which have an excess of zero outputs, are commonly encountered in problems such as climate or rare event modelling. Conventional machine learning approaches tend to overestimate the non-zeros leading to poor performance. We propose a novel model family of zero-inflated Gaussian processes (ZiGP) for such zero-inflated datasets, produced by sparse kernels through learning a latent probit Gaussian process that can zero out kernel rows and columns whenever the signal is absent. The ZiGPs are particularly useful for making the powerful Gaussian process networks more interpretable. We introduce sparse GP networks where variable-order latent modelling is achieved through sparse mixing signals. We derive the non-trivial stochastic variational inference tractably for scalable learning of the sparse kernels in both models. The novel output-sparse approach improves both prediction of zero-inflated data and interpretability of latent mixing models.

Motivation & Objective

• Address the challenge of modeling datasets with an excess of zero-valued observations, common in climate, ecology, and social sciences.
• Overcome the limitation of standard Gaussian processes in underestimating zeros and overestimating non-zero values in such data.
• Develop a novel Bayesian framework that combines a latent GP with a probit-linked switching process to induce exact zero predictions.
• Improve interpretability of Gaussian process networks by introducing sparse, variable-order mixing of latent signals through on-off processes in the mixing weights.
• Enable scalable learning via tractable stochastic variational inference for complex, high-dimensional sparse covariance structures in both ZiGP and sGPRN models.

Proposed method

• Propose a zero-inflated GP model where the covariance matrix is sparsified via a latent probit Gaussian process that determines when to zero out rows and columns.
• Model the sparsity in the kernel using a smooth, non-parametric probit link function over latent inputs, enabling continuous switching between 'on' and 'off' states.
• Derive a novel stochastic variational inference (SVI) framework for the intractable expectations of GP covariances under probit-linked sparsity, enabling scalable learning.
• Extend the SVI framework to conventional Gaussian process networks (GPRN), moving beyond diagonalized mean-field approximations to fully account for covariances.
• Introduce sparse GPRNs where mixing weights are governed by a separate on-off process, resulting in variable-order mixtures that activate only relevant latent functions per input location.
• Implement the inference using inducing points on spatial and temporal grids, with shared locations for support and weight functions, and use gamma priors on length-scales to encourage smoothness.

Experimental results

Research questions

• RQ1 How can Gaussian processes be adapted to model datasets with an excess of zero-valued observations without overestimating non-zero values?
• RQ2 Can a latent probit process be used to stochastically sparsify the kernel covariance matrix in a way that enables exact zero predictions in GP models?
• RQ3 What is a tractable inference method for learning the parameters of a zero-inflated GP with non-linear, non-Gaussian sparsity in the kernel?
• RQ4 How can sparse mixing in Gaussian process networks improve model interpretability and reduce complexity by enabling variable-order mixtures of latent functions?
• RQ5 Can stochastic variational inference be extended to handle full, non-diagonal covariance structures in sparse GPRNs with probit-linked sparsity?

Key findings

• The zero-inflated GP (ZiGP) model achieves significantly better performance on the precipitation dataset, with a higher number of perfect zero predictions compared to standard GP regression.
• On the rain dataset, ZiGP achieves the lowest mean squared error (MSE) and the highest F1 score, outperforming standard GP models in both regression accuracy and handling class imbalance.
• The sparse GPRN (sGPRN) model with 5 inducing points achieves lower RMSE (0.728) and MAE (0.567) than the baseline GPRN (RMSE: 0.732, MAE: 0.572), demonstrating improved regression performance.
• The sGPRN model learns interpretable sparsity patterns: smooth 'off' regions in the mixing weights indicate that only one of the two latent functions is used in certain spatial regions, reducing model complexity.
• The model with 10 inducing points also shows consistent improvement, with sGPRN achieving lower RMSE (0.749) and MAE (0.573) than GPRN (RMSE: 0.774, MAE: 0.586), confirming scalability and robustness.
• Visualization of the optimized sGPRN model shows that the support function (on-off process) and weight functions are spatially smooth, with distinct black (on) and white (off) regions, confirming adaptive, location-specific activation of latent functions.