[Paper Review] Residual Flows for Invertible Generative Modeling

Ricky T. Q. Chen, Jens Behrmann+2 more|arXiv (Cornell University)|Jun 6, 2019

Generative Adversarial Networks and Image Synthesis46 references96 citations

TL;DR

This paper introduces Residual Flows, an unbiased, memory-efficient flow-based model using invertible residual networks with a Russian roulette estimator, achieving competitive density estimation and strong generative/discriminative performance.

ABSTRACT

Flow-based generative models parameterize probability distributions through an invertible transformation and can be trained by maximum likelihood. Invertible residual networks provide a flexible family of transformations where only Lipschitz conditions rather than strict architectural constraints are needed for enforcing invertibility. However, prior work trained invertible residual networks for density estimation by relying on biased log-density estimates whose bias increased with the network's expressiveness. We give a tractable unbiased estimate of the log density using a "Russian roulette" estimator, and reduce the memory required during training by using an alternative infinite series for the gradient. Furthermore, we improve invertible residual blocks by proposing the use of activation functions that avoid derivative saturation and generalizing the Lipschitz condition to induced mixed norms. The resulting approach, called Residual Flows, achieves state-of-the-art performance on density estimation amongst flow-based models, and outperforms networks that use coupling blocks at joint generative and discriminative modeling.

Motivation & Objective

• Motivate a flexible, invertible architecture for density estimation without restrictive Jacobian structures.
• Provide an unbiased log-density estimator to enable maximum likelihood training with expressive networks.
• Reduce memory usage during training via gradient techniques and backpropagation tricks.
• Improve activation and Lipschitz control to prevent derivative saturation and broaden feasible norms.

Proposed method

• Adopt invertible residual networks with Lipschitz constraints to enable flexible, free-form Jacobians.
• Derive an unbiased log-density estimator using a Russian roulette-type randomized truncation of the log-determinant series.
• Use Skilling-Hutchinson trace estimation within the unbiased estimator to compute Jacobian traces.
• Introduce Neumann-series-based gradients to reduce memory during backpropagation through the log-determinant term.
• Propose LipSwish activation to avoid derivative saturation while maintaining a Lipschitz bound.
• Generalize Lipschitz constraints to induced mixed norms and learn norm orders to improve performance.

Experimental results

Research questions

• RQ1 Can an unbiased estimator enable maximum likelihood training for highly expressive invertible residual networks?
• RQ2 How can memory-efficient backpropagation be achieved for log-determinant terms in flow-based models?
• RQ3 Do Lipschitz-constrained residual blocks with suitable activations improve density estimation and sample quality compared to coupling-based flows?
• RQ4 Can mixed-norm Lipschitz constraints be effectively learned and do they improve performance across datasets?

Key findings

• Residual Flows achieve competitive or superior density estimation performance compared to state-of-the-art flow models on MNIST, CIFAR-10, ImageNet downsampled, and CelebA-HQ.
• An unbiased log-density estimator with a Russian roulette style truncation enables true maximum likelihood training for expressive i-ResNet-like architectures.
• Memory-efficient backpropagation techniques (Neumann gradient series and backward-in-forward gradients) significantly reduce peak memory usage.
• LipSwish activation reduces derivative saturation and improves training stability and performance under high Lipschitz constraints.
• In hybrid modeling, residual blocks outperform coupling blocks for joint generative-discriminative tasks, including MNIST and CIFAR-10.