[Paper Review] Scalable Variational Gaussian Process Regression Networks

Shibo Li, Wei Xing+2 more|arXiv (Cornell University)|Mar 25, 2020

Gaussian Processes and Bayesian Inference15 references4 citations

TL;DR

This paper proposes a scalable variational inference algorithm for Gaussian Process Regression Networks (GPRN) that captures complex posterior dependencies among latent functions and outputs using tensor-normal and matrix-normal variational posteriors. By leveraging tensorization of the output space and exploiting Kronecker structures in the evidence lower bound, the method achieves linear scaling with respect to data size, output dimension, and number of latent functions, significantly improving both inference quality and computational efficiency on large-scale multi-output regression tasks, including million-dimensional output prediction in physical simulations.

ABSTRACT

Gaussian process regression networks (GPRN) are powerful Bayesian models for multi-output regression, but their inference is intractable. To address this issue, existing methods use a fully factorized structure (or a mixture of such structures) over all the outputs and latent functions for posterior approximation, which, however, can miss the strong posterior dependencies among the latent variables and hurt the inference quality. In addition, the updates of the variational parameters are inefficient and can be prohibitively expensive for a large number of outputs. To overcome these limitations, we propose a scalable variational inference algorithm for GPRN, which not only captures the abundant posterior dependencies but also is much more efficient for massive outputs. We tensorize the output space and introduce tensor/matrix-normal variational posteriors to capture the posterior correlations and to reduce the parameters. We jointly optimize all the parameters and exploit the inherent Kronecker product structure in the variational model evidence lower bound to accelerate the computation. We demonstrate the advantages of our method in several real-world applications.

Motivation & Objective

• To address the limitations of existing GPRN inference methods, which use fully factorized variational posteriors that miss strong posterior dependencies among latent variables and weights.
• To reduce the computational cost of variational inference in GPRN, especially for large output dimensions D, where existing methods scale poorly with O(NK²D) or O(QN²KD) complexity.
• To develop a scalable and efficient inference algorithm that jointly optimizes all variational parameters while capturing rich posterior correlations in both latent functions and projection weights.
• To enable practical application of GPRN to real-world large-scale multi-output regression problems, such as MRI prediction and physical simulations with millions of outputs.

Proposed method

• Tensorize the output space by reshaping the D×K projection matrix into a tensor to exploit structural correlations in the outputs.
• Introduce a tensor-normal distribution as a joint variational posterior for the projection weights, which captures posterior dependencies while reducing the number of covariance parameters from O(D²) to O(D) in the tensor format.
• Use a matrix-normal distribution as the variational posterior for latent function values to model their joint posterior dependencies efficiently.
• Formulate the variational evidence lower bound (ELBO) in a form that allows decomposition using Kronecker product properties to accelerate log determinant and matrix inverse computations.
• Jointly optimize all variational parameters using gradient-based optimization, avoiding inefficient alternating updates.
• Leverage the Kronecker structure in the Hessian and Fisher information matrix approximations to achieve O(NKD) time complexity, linear in N, D, and K.

Experimental results

Research questions

• RQ1 Can a variational inference method for GPRN better capture posterior dependencies among latent functions and projection weights compared to fully factorized approximations?
• RQ2 Can the computational complexity of GPRN inference be reduced to scale linearly with the number of outputs D, especially when D is in the millions?
• RQ3 Does the proposed method maintain high predictive accuracy while achieving significant speedups on large-scale multi-output regression tasks?
• RQ4 Can the method be effectively applied to real-world problems with massive output dimensions, such as physical simulations with one million output variables?

Key findings

• On three benchmark datasets (Jura, Equity, PM2.5), the proposed SGPRN method achieved significantly lower mean absolute error (MAE) than both mean-field variational Bayes (MFVB) and nonparametric variational inference (NPV), with p-values < 0.05, demonstrating superior predictive performance.
• With 50 latent functions on the PM2.5 dataset, SGPRN achieved a 200× speedup over MFVB and a 785× speedup over NPV, confirming its linear O(NKD) complexity and scalability.
• On the Cantilever and GeneExp datasets, SGPRN consistently outperformed PCA-GP, KPCA-GP, and IsoMAP-GP in normalized root-mean-square error (NRMSE), except in one case where it was slightly worse than PCA-PG, indicating strong generalization on structured output correlations.
• In a large-scale physical simulation of lid-driven cavity flows with one million output dimensions, SGPRN achieved significantly lower NRMSE than all baselines, especially with 5 and 10 latent functions, proving its effectiveness in extreme-scale applications.
• The method's performance was robust across different tensorization schemes and number of latent functions, with consistent improvements in both accuracy and efficiency.