[Paper Review] Normalizing Flows for Probabilistic Modeling and Inference

George Papamakarios, Eric Nalisnick+3 more|arXiv (Cornell University)|Dec 5, 2019

Gaussian Processes and Bayesian Inference133 references343 citations

TL;DR

This paper provides a comprehensive review of normalizing flows, detailing their definition, expressiveness, and design principles, and surveys their use in probabilistic modeling, inference, and learning.

ABSTRACT

Normalizing flows provide a general mechanism for defining expressive probability distributions, only requiring the specification of a (usually simple) base distribution and a series of bijective transformations. There has been much recent work on normalizing flows, ranging from improving their expressive power to expanding their application. We believe the field has now matured and is in need of a unified perspective. In this review, we attempt to provide such a perspective by describing flows through the lens of probabilistic modeling and inference. We place special emphasis on the fundamental principles of flow design, and discuss foundational topics such as expressive power and computational trade-offs. We also broaden the conceptual framing of flows by relating them to more general probability transformations. Lastly, we summarize the use of flows for tasks such as generative modeling, approximate inference, and supervised learning.

Motivation & Objective

• Present a unified perspective on normalizing flows by connecting their design to probabilistic modeling and inference.
• Characterize the expressive power of flow-based models and analyze computational trade-offs.
• Relate flows to general probability transformations and discuss extensions to structured domains.
• Summarize core applications including generative modeling, approximate inference, and supervised learning.

Proposed method

• Define normalizing flows as invertible, differentiable transformations of a base distribution to model complex densities.
• Explain density computation via change of variables and the role of the Jacobian determinant.
• Describe building complex flows by composing simple, tractable transformations (finite flows) and discuss forward/inverse passes and Jacobian accumulation.
• Discuss training objectives such as forward KL (maximum likelihood) and reverse KL, including Monte Carlo gradient estimators.
• Outline alternative divergences (f-divergences, IPMs) and their implications for training implicit probabilistic models.
• Relate the forward and reverse KL viewpoints via changes of variables between x-space and u-space and present key equalities (KL equivalences).
• Provide a historical overview and categorize flow-construction approaches (e.g., autoregressive, planar, spline-based, coupling layers) with emphasis on tractable Jacobians.

Experimental results

Research questions

• RQ1 What conditions ensure the universal expressiveness of flow-based models relative to the target distribution?
• RQ2 How can we construct flows that are both highly expressive and computationally tractable for sampling and density evaluation?
• RQ3 How do different training objectives (forward vs reverse KL, other divergences) relate in the context of flows?
• RQ4 How do flows connect to broader probability transformations and what are their extensions to structured domains and geometries?
• RQ5 What are the typical applications of normalizing flows in generative modeling, approximate inference, and supervised learning?

Key findings

• Normalizing flows can represent a wide range of target distributions by composing simple diffeomorphisms with tractable Jacobians.
• Density evaluation and sampling hinge on the invertibility and tractable Jacobian of each transformation in the flow.
• Forward KL (maximum likelihood) training aligns the model to target samples, while reverse KL training aligns the base-induced distribution to the target, with a formal equivalence between these perspectives.
• The density of the base and the transformation jointly determine the model density, enabling training even when parts of the target are intractable.
• Alternative divergences (f-divergences and IPMs) offer flexible training options, including adversarial and variational approaches.
• Equivalences between training in x-space and u-space provide a unified view of flow optimization and connect flows to implicit probabilistic modeling.