[Paper Review] Stochastic Backpropagation and Approximate Inference in Deep Generative Models

Danilo Jimenez Rezende, Shakir Mohamed+1 more|arXiv (Cornell University)|Jan 16, 2014

Gaussian Processes and Bayesian Inference21 references2,647 citations

TL;DR

The paper derives stochastic backpropagation rules for optimizing deep generative models with Gaussian latent variables, presents Gaussian gradient identities, and discusses variance reduction and approximate inference techniques.

ABSTRACT

We marry ideas from deep neural networks and approximate Bayesian inference to derive a generalised class of deep, directed generative models, endowed with a new algorithm for scalable inference and learning. Our algorithm introduces a recognition model to represent approximate posterior distributions, and that acts as a stochastic encoder of the data. We develop stochastic back-propagation -- rules for back-propagation through stochastic variables -- and use this to develop an algorithm that allows for joint optimisation of the parameters of both the generative and recognition model. We demonstrate on several real-world data sets that the model generates realistic samples, provides accurate imputations of missing data and is a useful tool for high-dimensional data visualisation.

Motivation & Objective

• Clarify how stochastic backpropagation operates in deep generative models by separating deterministic and stochastic components.
• Derive Bonnet’s and Price’s theorems for Gaussian expectations to enable gradient computation.
• Provide practical methods for variance reduction and flexible coordinate transformations for non-Gaussian distributions.
• Illustrate probabilistic inference and learning in deep directed models with Gaussian latent structure.

Proposed method

• Formalize joint log-likelihood by separating deterministic and stochastic parts through a coordinate change from h_l to ξ_l.
• Derive Gaussian gradient identities (Bonnet’s and Price’s theorems) for efficient gradient estimation.
• Present two stochastic backpropagation approaches: a product-rule-based method with a B(x;θ) transform and an alternative coordinate-transformation approach for various distributions.
• Discuss variance reduction techniques via control variates and relate REINFORCE to this framework.
• Describe estimators for marginal likelihood via importance sampling and outline missing data imputation procedures using a Markov chain on missing pixels.

Experimental results

Research questions

• RQ1 How can stochastic backpropagation be derived and applied to deep generative models with Gaussian latent variables?
• RQ2 What are the Gaussian gradient identities that enable efficient gradient estimation for expectations under Gaussian noise?
• RQ3 How can variance be reduced in stochastic gradient estimators used in variational learning?
• RQ4 How can we extend stochastic backpropagation to non-Gaussian distributions via coordinate transformations or nonlinear reparameterizations?
• RQ5 How can marginal likelihoods and missing data be handled within this framework?

Key findings

• Bonnet’s theorem shows the gradient of an expectation under a Gaussian with respect to the mean equals the expectation of the gradient of the function.
• Price’s theorem expresses the gradient of an expectation with respect to the covariance as half the expectation of the Hessian of the function under Gaussian noise.
• A product-rule based stochastic backpropagation approach introduces a nonlinear function B(x) to transform gradients and enable backpropagation across stochastic layers.
• Alternative coordinate-transformations allow stochastic backpropagation for distributions beyond the Gaussian, such as Levy, Log-Normal, and Generalized Extreme Value families.
• Variance analysis demonstrates that REINFORCE-type estimators can have higher variance than Bonnet/Prices-based approaches, especially as the number of variables grows.
• Importance sampling provides a practical estimator for marginal likelihood using samples from the recognition model.