[Paper Review] Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design

Jonathan Ho, Xi Chen+3 more|arXiv (Cornell University)|Feb 1, 2019

Generative Adversarial Networks and Image SynthesisComputer Science188 citations

TL;DR

Flow++ advances non-autoregressive flow models by introducing variational dequantization, logistic mixture coupling flows, and self-attention conditioning, achieving state-of-the-art density estimation among non-autoregressive models on CIFAR-10 and 32x32/64x64 ImageNet. It narrows the gap with autoregressive models while preserving fast sampling.

ABSTRACT

Flow-based generative models are powerful exact likelihood models with efficient sampling and inference. Despite their computational efficiency, flow-based models generally have much worse density modeling performance compared to state-of-the-art autoregressive models. In this paper, we investigate and improve upon three limiting design choices employed by flow-based models in prior work: the use of uniform noise for dequantization, the use of inexpressive affine flows, and the use of purely convolutional conditioning networks in coupling layers. Based on our findings, we propose Flow++, a new flow-based model that is now the state-of-the-art non-autoregressive model for unconditional density estimation on standard image benchmarks. Our work has begun to close the significant performance gap that has so far existed between autoregressive models and flow-based models. Our implementation is available at https://github.com/aravindsrinivas/flowpp

Motivation & Objective

• Motivate improvements to flow-based models that trade off density estimation, sampling speed, and tractable likelihood.
• Identify three limiting design choices in prior flow models: uniform dequantization, affine coupling, and purely convolutional conditioning.
• Propose Flow++ with variational dequantization, logistic mixture CDF coupling, and self-attention conditioning to boost density modeling performance.
• Demonstrate state-of-the-art density estimation for non-autoregressive models on CIFAR-10 and ImageNet, and analyze ablations to quantify each design contribution.

Proposed method

• Use variational dequantization where dequantization noise q(u|x) is modeled by a conditional flow q_x(u) and optimized jointly with the model p_model.
• Replace affine coupling with a expressive logistic mixture CDF coupling in the coupling layers.
• Incorporate self-attention in the conditioning networks that generate coupling parameters, via a gated residual network with attention blocks.
• Compute exact log-likelihood via invertible flow composition f = f1 ○ ... ○ fL and use tractable Jacobian determinants for training and sampling.
• Adopt a differentiable dequantization objective that improves training and generalization compared to uniform dequantization (lower bound gap).
• Evaluate Flow++ on CIFAR-10, 32x32 ImageNet, and 64x64 ImageNet, compare to non-autoregressive and autoregressive baselines, and perform ablations on dequantization, coupling, and conditioning.

Experimental results

Research questions

• RQ1 Can variational dequantization improve density estimation and generalization over uniform dequantization in flow models?
• RQ2 Does replacing affine coupling with logistic mixture CDF couplings enhance expressiveness without sacrificing tractability?
• RQ3 Does incorporating self-attention in conditioning networks improve the conditioning capacity and density modeling performance?
• RQ4 How close can non-autoregressive Flow++ get to autoregressive models on standard image benchmarks while maintaining fast sampling?

Key findings

• Flow++ achieves state-of-the-art density modeling among non-autoregressive models on CIFAR-10 and ImageNet (32x32 and 64x64).
• With 1 sample, Flow++ attains 3.12 bits/dim on CIFAR-10; training with variational dequantization reduces train-test gap compared to uniform dequantization.
• Ablations show variational dequantization costs about 0.127 bits/dim more than Flow++; switching from logistic mixture coupling to affine coupling costs about 0.03 bits/dim; removing self-attention also costs about 0.03 bits/dim.
• Flow++ samples are perceptually competitive with PixelCNN-type autoregressive models and achieve substantially faster sampling (e.g., CIFAR-10 sampling ~0.32 seconds for 8 samples on a single GPU).
• Flow++ results are competitive with early autoregressive models (PixelCNN family) and surpass prior non-autoregressive flows like RealNVP and Glow in density estimation on the tested benchmarks.