[Paper Review] On the expressivity of bi-Lipschitz normalizing flows

Alexandre Vérine, Benjamin Négrevergne+2 more|arXiv (Cornell University)|Jul 15, 2021

Image and Signal Denoising Methods16 references4 citations

TL;DR

This paper investigates the expressivity limitations of bi-Lipschitz normalizing flows by deriving novel lower bounds on the Total Variation (TV) distance between target distributions and their approximations. It shows that bi-Lipschitz constraints—especially bounded forward and inverse Lipschitz constants—can prevent accurate modeling of complex data distributions with high-density or low-density regions, and proposes improved latent distributions as a remedy.

ABSTRACT

An invertible function is bi-Lipschitz if both the function and its inverse have bounded Lipschitz constants. Nowadays, most Normalizing Flows are bi-Lipschitz by design or by training to limit numerical errors (among other things). In this paper, we discuss the expressivity of bi-Lipschitz Normalizing Flows and identify several target distributions that are difficult to approximate using such models. Then, we characterize the expressivity of bi-Lipschitz Normalizing Flows by giving several lower bounds on the Total Variation distance between these particularly unfavorable distributions and their best possible approximation. Finally, we discuss potential remedies which include using more complex latent distributions.

Motivation & Objective

• To analyze the expressivity limitations of bi-Lipschitz normalizing flows in approximating complex data distributions.
• To identify specific data distribution characteristics—such as high-density or low-density regions—that challenge bi-Lipschitz models.
• To derive new theoretical lower bounds on the Total Variation distance between target and approximated distributions.
• To explore remedies, particularly using more complex latent distributions like Gaussian Mixture Models, to mitigate expressivity constraints.
• To provide a theoretical framework applicable to any data distribution without restrictive assumptions on support topology.

Proposed method

• Derives a general lower bound on TV distance based on the forward mapping's Lipschitz constant $L_1$, applicable to any data distribution.
• Establishes a second lower bound dependent on the inverse mapping's Lipschitz constant $L_2$, highlighting limitations in modeling low-density regions.
• Introduces a third bound that depends on both $L_1$ and $L_2$, offering a complementary perspective on expressivity trade-offs.
• Adapts the bounds to Gaussian Mixture latent distributions by incorporating learnable means and variances $\sigma_i$ and number of modes $K$.
• Analyzes the impact of $K$ and $\sigma_i$ on the lower bound, showing that increasing $K$ or learning $\sigma_i$ can mitigate approximation errors.
• Discusses the challenges of training mixture-based models and continuous partitioning methods, noting their complexity and instability.

Experimental results

Research questions

• RQ1 How do bi-Lipschitz constraints on both forward and inverse mappings limit the ability of normalizing flows to approximate complex data distributions?
• RQ2 What are the theoretical lower bounds on the Total Variation distance between a target distribution and its approximation under bi-Lipschitz constraints?
• RQ3 How do high-density and low-density regions in the data space affect the approximation error in bi-Lipschitz normalizing flows?
• RQ4 Can using a Gaussian Mixture latent distribution reduce the lower bound on TV distance and improve expressivity?
• RQ5 What are the practical and theoretical challenges in training models with complex latent distributions to overcome bi-Lipschitz limitations?

Key findings

• The paper establishes a lower bound on the Total Variation distance that depends on the forward mapping's Lipschitz constant $L_1$, showing that dense data regions can lead to unavoidable approximation errors.
• A second lower bound is derived based on the inverse mapping's Lipschitz constant $L_2$, indicating that low-density regions between high-density clusters also induce approximation errors.
• A third bound is introduced that depends on both $L_1$ and $L_2$, revealing a trade-off: improving one can worsen the other, without a general improvement in expressivity.
• When using a Gaussian Mixture latent distribution with $K$ modes and learnable variances $\sigma_i$, the lower bound increases with $K$ but can be compensated by learning $\sigma_i$, suggesting a path to improved approximation.
• Despite theoretical promise, practical training of mixture-based or partitioned normalizing flows remains challenging due to instability and lack of efficient methods to set hyperparameters like $K$.
• The results imply that bi-Lipschitz constraints inherently limit expressivity for certain data distributions, and that remedies require careful architectural and training design.