[Paper Review] Variational Inference with Normalizing Flows

Danilo Jimenez Rezende, Shakir Mohamed|arXiv (Cornell University)|May 21, 2015

Model Reduction and Neural Networks34 references1,425 citations

TL;DR

This paper proposes using normalizing flows to construct highly flexible, scalable approximate posterior distributions in variational inference, enabling richer posterior approximations than traditional mean-field or structured methods. By applying invertible, autoregressive transformations to a simple base density, the method achieves tighter variational bounds and systematically improves test likelihoods on MNIST and CIFAR-10, with performance improving monotonically with flow depth.

ABSTRACT

The choice of approximate posterior distribution is one of the core problems in variational inference. Most applications of variational inference employ simple families of posterior approximations in order to allow for efficient inference, focusing on mean-field or other simple structured approximations. This restriction has a significant impact on the quality of inferences made using variational methods. We introduce a new approach for specifying flexible, arbitrarily complex and scalable approximate posterior distributions. Our approximations are distributions constructed through a normalizing flow, whereby a simple initial density is transformed into a more complex one by applying a sequence of invertible transformations until a desired level of complexity is attained. We use this view of normalizing flows to develop categories of finite and infinitesimal flows and provide a unified view of approaches for constructing rich posterior approximations. We demonstrate that the theoretical advantages of having posteriors that better match the true posterior, combined with the scalability of amortized variational approaches, provides a clear improvement in performance and applicability of variational inference.

Motivation & Objective

• To address the limitation in variational inference where simple posterior approximations (e.g., mean-field) restrict model accuracy and fail to capture complex posterior structure.
• To develop a scalable and flexible framework for posterior approximation that can match the complexity of true posteriors.
• To unify existing approaches for rich posterior estimation under the normalizing flow formalism.
• To demonstrate that increasing flow depth leads to systematically better posterior approximations and improved likelihoods on benchmark datasets.

Proposed method

• Uses normalizing flows—sequences of invertible, differentiable transformations—to transform a simple base density (e.g., Gaussian) into a complex, flexible posterior approximation.
• Employs amortized variational inference with inference networks to parameterize the flow transformations, enabling efficient, scalable training.
• Derives a modified variational lower bound that includes a log-Jacobian determinant term to ensure valid probability density under the transformation.
• Introduces infinitesimal flows via continuous normalizing flows, enabling asymptotic convergence to the true posterior under sufficient depth.
• Applies flow transformations using coupling layers (e.g., RealNVP-style) to maintain computational efficiency and invertibility.
• Uses Monte Carlo gradient estimation for end-to-end training of the flow-based posterior approximations.

Experimental results

Research questions

• RQ1 Can normalizing flows provide a scalable and flexible alternative to mean-field and structured variational approximations in posterior estimation?
• RQ2 Does increasing the depth of a normalizing flow lead to systematically better posterior approximations and improved model likelihoods?
• RQ3 Can infinitesimal normalizing flows asymptotically recover the true posterior distribution, overcoming a key limitation of classical variational inference?
• RQ4 How do flow-based posteriors compare to other advanced methods like NICE, HVI, or DARN in terms of test likelihood and inference quality?

Key findings

• On binarized MNIST, increasing flow depth from K=10 to K=80 reduced test set negative log-likelihood from ≤87.5 to ≤85.1, demonstrating consistent improvement with complexity.
• The DLGM+NF model with K=80 achieved a test negative log-likelihood of ≤85.1, outperforming the NICE-based baseline (≤87.2) and the HVI baseline (85.51 with 8 leapfrog steps).
• On CIFAR-10, increasing flow depth from K=0 to K=10 reduced test negative log-likelihood from -293.7 to -320.7, showing strong monotonic improvement.
• The method with K=10 flow steps achieved a test likelihood of -320.7 on CIFAR-10, outperforming the DARN model with adaptive noise (84.13) when using a comparable model architecture.
• Theoretical analysis shows that infinitesimal flows can, in the asymptotic regime, recover the true posterior, resolving a long-standing criticism of variational inference.
• The framework unifies various approaches to flexible posterior approximation under a single normalizing flow formalism, revealing shared structural principles.