[Paper Review] Doubly Semi-Implicit Variational Inference

Dmitry Molchanov, В. А. Харитонов+2 more|arXiv (Cornell University)|Oct 5, 2018

Gaussian Processes and Bayesian Inference34 references4 citations

TL;DR

This paper introduces Doubly Semi-Implicit Variational Inference (DSIVI), a framework for variational inference when both the approximate posterior and prior are semi-implicit distributions—defined as mixtures of tractable densities with flexible, implicit mixing measures. DSIVI provides an asymptotically exact lower bound on the ELBO and an upper bound that can be made arbitrarily tight, enabling accurate variational learning in models with implicit priors and posteriors, and it improves VampPrior in VAEs by enabling a more expressive, learnable prior without warm-up.

ABSTRACT

We extend the existing framework of semi-implicit variational inference (SIVI) and introduce doubly semi-implicit variational inference (DSIVI), a way to perform variational inference and learning when both the approximate posterior and the prior distribution are semi-implicit. In other words, DSIVI performs inference in models where the prior and the posterior can be expressed as an intractable infinite mixture of some analytic density with a highly flexible implicit mixing distribution. We provide a sandwich bound on the evidence lower bound (ELBO) objective that can be made arbitrarily tight. Unlike discriminator-based and kernel-based approaches to implicit variational inference, DSIVI optimizes a proper lower bound on ELBO that is asymptotically exact. We evaluate DSIVI on a set of problems that benefit from implicit priors. In particular, we show that DSIVI gives rise to a simple modification of VampPrior, the current state-of-the-art prior for variational autoencoders, which improves its performance.

Motivation & Objective

• To address the limitation of existing variational inference methods that require tractable densities for both prior and posterior, especially when implicit distributions are needed for better modeling.
• To develop a framework that supports variational inference when both the approximate posterior and the prior are semi-implicit, i.e., mixtures of analytic densities with implicit mixing distributions.
• To provide a lower bound on the ELBO that is asymptotically exact and an upper bound that can be made arbitrarily tight, enabling reliable optimization.
• To demonstrate the effectiveness of DSIVI in practical settings such as VAEs with implicit priors, particularly improving upon state-of-the-art methods like VampPrior.

Proposed method

• Proposes a doubly semi-implicit variational inference (DSIVI) framework where both the prior and posterior are modeled as semi-implicit distributions—mixtures of analytic densities with implicit mixing measures.
• Derives a SIVI-inspired asymptotically exact lower bound on the ELBO using reparameterized gradients and the variational representation of KL divergence.
• Constructs an upper bound on the ELBO using a sandwich bound that becomes arbitrarily tight with increasing sample size.
• Employs a reparameterization trick for gradient estimation by expressing samples from the posterior and prior as deterministic transformations of noise variables.
• Uses a neural network to parameterize the mixing distribution (e.g., for the prior), allowing end-to-end training of the prior and posterior jointly.
• Applies the method to VAEs by modeling the prior as a semi-implicit mixture of Gaussians, with learnable pseudo-inputs and a flexible mixing network.

Experimental results

Research questions

• RQ1 Can we perform variational inference when both the prior and posterior are semi-implicit, i.e., defined via implicit mixing distributions?
• RQ2 Can we construct a lower bound on the ELBO that is asymptotically exact in such settings?
• RQ3 Can we achieve tighter bounds on the ELBO by sandwiching it between a lower and upper bound that can be made arbitrarily tight?
• RQ4 Does DSIVI improve performance in models like VAEs when used with implicit priors, especially compared to existing methods like VampPrior?

Key findings

• DSIVI achieves a tighter lower bound on the ELBO than previous implicit variational inference methods, with the bound becoming asymptotically exact as sample size increases.
• The method enables end-to-end training of both the prior and posterior in a semi-implicit framework, allowing for more expressive modeling than standard VampPrior.
• In VAEs on MNIST, DSIVI-agg (semi-implicit aggregated posterior) achieves a test log-likelihood of ≥−82.16 with K=5000, outperforming VampPrior (−82.38) and VampPrior-data (−85.05).
• DSIVI-prior, which uses a fully factorized Gaussian conditional with a neural network mixing distribution, achieves −82.27 log-likelihood without warm-up, outperforming VampPrior.
• For hierarchical VAEs, DSIVI-agg achieves ≥−81.09 log-likelihood, surpassing VampPrior (−81.24) and VampPrior-data (−81.71).
• The framework successfully learns complex target distributions even when both prior and posterior are semi-implicit, as shown in sequential approximation experiments with Gaussian mixtures.