[Paper Review] You say Normalizing Flows I see Bayesian Networks

Antoine Wehenkel, Gilles Louppe|arXiv (Cornell University)|Jun 1, 2020

Gaussian Processes and Bayesian Inference4 references4 citations

TL;DR

This paper reinterprets normalizing flows as Bayesian networks, revealing that coupling and autoregressive flows correspond to specific BN topologies with learnable node densities. It demonstrates that stacking more than three affine transformation layers triggers a fundamental capacity leap, but affine normalizing flows remain non-universal density approximators regardless of depth, due to structural constraints on modeling non-normal components.

ABSTRACT

Normalizing flows have emerged as an important family of deep neural networks for modelling complex probability distributions. In this note, we revisit their coupling and autoregressive transformation layers as probabilistic graphical models and show that they reduce to Bayesian networks with a pre-defined topology and a learnable density at each node. From this new perspective, we provide three results. First, we show that stacking multiple transformations in a normalizing flow relaxes independence assumptions and entangles the model distribution. Second, we show that a fundamental leap of capacity emerges when the depth of affine flows exceeds 3 transformation layers. Third, we prove the non-universality of the affine normalizing flow, regardless of its depth.

Motivation & Objective

• To reinterpret normalizing flows through the lens of probabilistic graphical models, specifically Bayesian networks.
• To analyze how stacking transformation layers affects independence assumptions and model capacity in normalizing flows.
• To investigate the universality of affine normalizing flows as density approximators.
• To provide theoretical insight into the structural limitations and capacity dynamics of normalizing flow architectures.

Proposed method

• Reformulates coupling and autoregressive normalizing flow layers as Bayesian networks with fixed topologies and learnable conditional densities at each node.
• Uses the change of variables formula to express the joint density of the transformed variables in terms of the base distribution and Jacobian determinants.
• Applies d-separation and I-map theory to analyze conditional independencies in the equivalent Bayesian network representations.
• Constructs equivalent Bayesian networks for single- and multi-step flows, showing how dependencies are introduced with each layer.
• Employs a counterexample argument to prove non-universality by showing affine flows cannot model non-normal marginal distributions when components have limited latent ancestors.
• Analyzes the structural constraints of bijective transformations, proving that removing undirected edges from the BN would violate bijectivity.

Experimental results

Research questions

• RQ1 How can normalizing flow architectures be formally mapped to Bayesian network structures?
• RQ2 What happens to conditional independence assumptions as more flow layers are stacked?
• RQ3 Does a critical depth exist beyond which normalizing flow capacity increases significantly?
• RQ4 Can affine normalizing flows universally approximate any probability distribution, regardless of depth?
• RQ5 What structural limitations prevent affine normalizing flows from achieving universal approximation?

Key findings

• Stacking multiple normalizing flow transformations relaxes independence assumptions and entangles the model distribution by introducing new conditional dependencies.
• A fundamental capacity leap occurs when the depth of affine normalizing flows exceeds three layers, indicating a qualitative change in modeling capability.
• Affine normalizing flows are not universal density approximators, even with infinite depth, due to structural constraints on modeling non-normal marginal distributions.
• When a component of the data vector is marginally non-normal, the equivalent Bayesian network cannot represent it unless the latent ancestor structure allows non-linear transformations, which is restricted by bijectivity.
• The model's capacity is limited by the fact that any component of the data vector must be a linear function of its latent ancestors if it has only one, leading to normality constraints.
• The equivalence between normalizing flows and Bayesian networks reveals that additional flow layers do not improve capacity beyond a certain point when conditioner capacity is already sufficient, as seen empirically in 2D toy problems.