[Paper Review] Implicit Posterior Variational Inference for Deep Gaussian Processes

Haibin Yu, Yizhou Chen+3 more|arXiv (Cornell University)|Oct 26, 2019

Gaussian Processes and Bayesian Inference15 citations

TL;DR

This paper proposes Implicit Posterior Variational Inference (IPVI) for Deep Gaussian Processes (DGPs), framing DGP inference as a two-player game to achieve unbiased posterior belief estimation with time efficiency. By leveraging best-response dynamics to find a Nash equilibrium that corresponds to an unbiased posterior, IPVI outperforms state-of-the-art methods in regression, classification, and unsupervised learning tasks.

ABSTRACT

A multi-layer deep Gaussian process (DGP) model is a hierarchical composition of GP models with a greater expressive power. Exact DGP inference is intractable, which has motivated the recent development of deterministic and stochastic approximation methods. Unfortunately, the deterministic approximation methods yield a biased posterior belief while the stochastic one is computationally costly. This paper presents an implicit posterior variational inference (IPVI) framework for DGPs that can ideally recover an unbiased posterior belief and still preserve time efficiency. Inspired by generative adversarial networks, our IPVI framework achieves this by casting the DGP inference problem as a two-player game in which a Nash equilibrium, interestingly, coincides with an unbiased posterior belief. This consequently inspires us to devise a best-response dynamics algorithm to search for a Nash equilibrium (i.e., an unbiased posterior belief). Empirical evaluation shows that IPVI outperforms the state-of-the-art approximation methods for DGPs.

Motivation & Objective

• Address the limitation of deterministic DGP approximations that produce biased posterior beliefs due to restrictive Gaussian assumptions.
• Overcome the computational inefficiency of stochastic methods like SGHMC, which rely on sequential sampling and slow convergence.
• Develop a time-efficient method that achieves unbiased posterior inference for DGPs, combining the strengths of both deterministic and stochastic approaches.
• Enable accurate modeling of complex, multi-modal posterior distributions in unsupervised and supervised learning settings.
• Design a parameter-tying generator architecture to reduce overfitting and accelerate training and prediction in DGP models.

Proposed method

• Cast DGP inference as a two-player game between a generator and a discriminator, where the Nash equilibrium corresponds to an unbiased posterior belief of the inducing variables.
• Use best-response dynamics to iteratively update the generator and discriminator, converging toward a Nash equilibrium that represents the true posterior.
• Design a generator with parameter tying across DGP layers to enforce structural consistency and reduce overfitting.
• Leverage implicit generative modeling inspired by GANs to represent complex, non-Gaussian posteriors without explicit likelihood computation.
• Train the generator to produce samples that match the true posterior distribution by minimizing a divergence measure via adversarial training.
• Apply the framework to both supervised (regression, classification) and unsupervised (FreyFace reconstruction) tasks using variational posterior approximation.

Experimental results

Research questions

• RQ1 Can a game-theoretic framework be used to achieve unbiased posterior inference in Deep Gaussian Processes without the computational cost of MCMC sampling?
• RQ2 Does a Nash equilibrium in the proposed two-player game correspond to an unbiased posterior belief of the inducing variables in DGPs?
• RQ3 Can implicit posterior variational inference via adversarial training outperform existing deterministic and stochastic DGP approximation methods in predictive accuracy?
• RQ4 How well can the proposed IPVI framework model complex, multi-modal posterior distributions in unsupervised representation learning tasks?
• RQ5 To what extent does parameter tying in the generator architecture improve generalization and reduce overfitting in DGP models?

Key findings

• IPVI achieves state-of-the-art performance on regression and classification benchmarks, outperforming both deterministic and stochastic baselines.
• On the FreyFace dataset, IPVI accurately captures the multi-modal nature of the posterior distribution, recovering both calm and happy facial expressions from partial observations.
• The Gaussian approximation fails to model multi-modal posteriors, capturing only one mode (e.g., mode A), while IPVI successfully recovers both modes (A and B).
• Empirical results show that IPVI with parameter tying achieves better generalization and faster training and prediction compared to models without tying.
• The best-response dynamics algorithm converges to a Nash equilibrium that corresponds to an unbiased posterior belief, validating the theoretical foundation of the framework.
• In supervised tasks, IPVI achieves higher test log-likelihood and lower RMSE than competing methods, especially in settings with non-Gaussian posteriors.