[Paper Review] Sliced Iterative Normalizing Flows

Biwei Dai, Uroš Seljak|arXiv (Cornell University)|Jul 1, 2020

Generative Adversarial Networks and Image Synthesis59 references4 citations

TL;DR

This paper proposes Sliced Iterative Normalizing Flows (SINF), a novel deep generative model that uses iterative Optimal Transport on 1D slices to transform a simple prior into complex data distributions. By selecting slicing directions to maximize Wasserstein distance at each step, SINF achieves state-of-the-art sample quality comparable to GANs and competitive density estimation with high stability and low hyperparameter sensitivity, even on small datasets.

ABSTRACT

We develop an iterative (greedy) deep learning (DL) algorithm which is able to transform an arbitrary probability distribution function (PDF) into the target PDF. The model is based on iterative Optimal Transport of a series of 1D slices, matching on each slice the marginal PDF to the target. The axes of the orthogonal slices are chosen to maximize the PDF difference using Wasserstein distance at each iteration, which enables the algorithm to scale well to high dimensions. As special cases of this algorithm, we introduce two sliced iterative Normalizing Flow (SINF) models, which map from the data to the latent space (GIS) and vice versa (SIG). We show that SIG is able to generate high quality samples of image datasets, which match the GAN benchmarks, while GIS obtains competitive results on density estimation tasks compared to the density trained NFs, and is more stable, faster, and achieves higher $p(x)$ when trained on small training sets. SINF approach deviates significantly from the current DL paradigm, as it is greedy and does not use concepts such as mini-batching, stochastic gradient descent and gradient back-propagation through deep layers.

Motivation & Objective

• Address the limitations of standard normalizing flows and GANs in sample quality and training stability by introducing a new optimization paradigm.
• Overcome the high-dimensional complexity of Optimal Transport by decomposing it into a sequence of 1D transport problems on adaptive slices.
• Develop a greedy, non-backpropagation-based training framework that avoids common deep learning pitfalls like vanishing gradients and mode collapse.
• Achieve competitive performance in both generative modeling and density estimation with minimal hyperparameter sensitivity and high robustness to random seeds.

Proposed method

• Propose a new metric, max K-sliced Wasserstein distance, to guide the selection of orthogonal slicing directions that maximize discrepancy between current and target distributions.
• Iteratively apply 1D Optimal Transport to marginal distributions along selected slicing directions, updating the transformation in a greedy, additive manner.
• Use rational quadratic splines for 1D coupling flows with regularization on spline parameters and linear extrapolation to ensure stability.
• Introduce a hierarchical patch-based architecture for high-dimensional images, starting from large patches and progressively refining with smaller ones.
• Apply logit transformation for preprocessing in density estimation tasks and use KDE-based bandwidth estimation via Scott’s rule for non-parametric density estimation.
• Train two variants: GIS (data-to-latent) for density estimation and SIG (latent-to-data) for sampling, both using iterative, non-backpropagation-based optimization.

Experimental results

Research questions

• RQ1 Can iterative 1D Optimal Transport on adaptive slices achieve high-quality sample generation comparable to GANs without adversarial training?
• RQ2 Can this approach achieve competitive density estimation performance on small datasets, outperforming standard normalizing flows?
• RQ3 Does the absence of backpropagation and mini-batching lead to significantly more stable training with less hyperparameter sensitivity?
• RQ4 Can the max K-sliced Wasserstein distance metric effectively guide the selection of slicing directions to scale efficiently to high dimensions?
• RQ5 How does the hierarchical patch-based architecture improve performance on high-dimensional image data like CIFAR-10 and CelebA?

Key findings

• SIG generates high-quality image samples that match GAN benchmarks in terms of FID and IS scores, demonstrating strong generative capability without adversarial training.
• GIS achieves competitive density estimation results on MNIST, CIFAR-10, and CelebA, with higher log-likelihoods than standard NFs, especially on small training sets.
• The model shows exceptional training stability: no training failures were observed across multiple random seeds and datasets.
• Performance is highly insensitive to hyperparameter choices, including learning rate, regularization, and kernel bandwidth, with results stable across different configurations.
• On small datasets (e.g., 1000 samples), GIS achieves significantly higher log-likelihood than baseline NFs, indicating superior generalization.
• The hierarchical patch-based approach with decreasing patch sizes (from q=8 to q=2) improves performance on image datasets, with 200 iterations per patch size yielding optimal results.