[Paper Review] Deep Gaussian Processes with Decoupled Inducing Inputs

Marton Havasi, José Miguel Hernández-Lobato+1 more|arXiv (Cornell University)|Jan 9, 2018

Gaussian Processes and Bayesian Inference3 references3 citations

TL;DR

This paper introduces decoupled inducing inputs in Deep Gaussian Processes (DGPs), using separate sets of inducing points for mean and variance computation. By reducing the number of inducing points for variance estimation while retaining more for mean estimation, the method achieves faster training and improved predictive performance without sacrificing uncertainty calibration.

ABSTRACT

Deep Gaussian Processes (DGP) are hierarchical generalizations of Gaussian Processes (GP) that have proven to work effectively on a multiple supervised regression tasks. They combine the well calibrated uncertainty estimates of GPs with the great flexibility of multilayer models. In DGPs, given the inputs, the outputs of the layers are Gaussian distributions parameterized by their means and covariances. These layers are realized as Sparse GPs where the training data is approximated using a small set of pseudo points. In this work, we show that the computational cost of DGPs can be reduced with no loss in performance by using a separate, smaller set of pseudo points when calculating the layerwise variance while using a larger set of pseudo points when calculating the layerwise mean. This enabled us to train larger models that have lower cost and better predictive performance.

Motivation & Objective

• To reduce the computational cost of training Deep Gaussian Processes (DGPs) without compromising predictive performance.
• To investigate whether decoupling inducing inputs for mean and variance computation—previously shown effective in standard GPs—can be extended to the deep GP setting.
• To improve model scalability by enabling larger DGP architectures with lower computational overhead.
• To evaluate whether the decoupled approach leads to better generalization and uncertainty estimation compared to standard DGPs.
• To address the challenge of vanishing gradients in inducing point optimization through improved parameterization.

Proposed method

• Adapts the decoupled GP framework (Cheng & Boots, 2017) to DGPs by using two distinct sets of inducing inputs: $\bm{Z}_a$ for mean computation and $\bm{Z}_b$ for variance computation.
• Replaces the standard GP mean and covariance computation with a dual parameterization: $\tilde{\mu} = \bm{K}_{\bm{X}\bm{Z}_a}\bm{a}$ and $\tilde{\Sigma} = \bm{K}_{\bm{X}\bm{X}} - \bm{K}_{\bm{X}\bm{Z}_b}(\bm{B}^{-1} + \bm{K}_{\bm{Z}_b\bm{Z}_b})^{-1}\bm{K}_{\bm{X}\bm{Z}_b}^T$.
• Maintains the variational inference framework for DGPs, with the ELBO adapted to account for separate inducing input sets in each layer.
• Uses a Cholesky decomposition for $\bm{S}$ and $\bm{B}$ to ensure numerical stability during optimization.
• Employs stochastic gradient descent (Adam) with mini-batches and a fixed learning rate of 0.01 over 5000 epochs.
• Applies a static mean function per layer to stabilize training and avoid degenerate covariance matrices.

Experimental results

Research questions

• RQ1 Can decoupling inducing inputs for mean and variance computation be successfully extended to Deep Gaussian Processes?
• RQ2 Does the decoupled DGP achieve better predictive performance than the standard DGP with equivalent computational cost?
• RQ3 Is the decoupled DGP more computationally efficient, particularly in deeper architectures?
• RQ4 How does the choice of inducing point set size for mean ($M_a$) versus variance ($M_b$) affect model performance and training time?
• RQ5 Does the decoupled approach mitigate issues such as vanishing gradients in DGP optimization?

Key findings

• The decoupled DGP achieved higher test log-likelihood than the standard DGP across all three datasets: kin8nm, protein, and molecules.
• The decoupled DGP achieved lower test root mean squared error (RMSE) than the standard DGP on all datasets except for the two-layer model on kin8nm.
• For models with at least one hidden layer, the decoupled DGP was faster to train, with median runtimes reduced by up to 30% on the molecules dataset.
• The decoupled model with $M_a = 500$, $M_b = 100$ had significantly lower runtime than the standard DGP with $M = 200$ for deeper models (e.g., 1017s vs. 1601s for $L=4$ on molecules).
• The performance gain was most pronounced in deeper architectures, where the computational savings from reducing $M_b$ were most impactful.
• The decoupled approach achieved a complexity bound of $O\big{(}L(DNM_a + M_a^3 + DNM_b^2 + M_b^3)\big{)}$, approaching the theoretical optimum for large batch sizes.