[Paper Review] Emerging Convolutions for Generative Normalizing Flows

Emiel Hoogeboom, Rianne van den Berg+1 more|arXiv (Cornell University)|Jan 30, 2019

Generative Adversarial Networks and Image Synthesis43 citations

TL;DR

The paper generalizes invertible convolutions to dxd forms via emerging and periodic convolutions, plus a QR-parametrized 1x1 convolution, and shows improved density modeling performance on CIFAR-10, ImageNet, and galaxy images.

ABSTRACT

Generative flows are attractive because they admit exact likelihood optimization and efficient image synthesis. Recently, Kingma & Dhariwal (2018) demonstrated with Glow that generative flows are capable of generating high quality images. We generalize the 1 x 1 convolutions proposed in Glow to invertible d x d convolutions, which are more flexible since they operate on both channel and spatial axes. We propose two methods to produce invertible convolutions that have receptive fields identical to standard convolutions: Emerging convolutions are obtained by chaining specific autoregressive convolutions, and periodic convolutions are decoupled in the frequency domain. Our experiments show that the flexibility of d x d convolutions significantly improves the performance of generative flow models on galaxy images, CIFAR10 and ImageNet.

Motivation & Objective

• Improve flexibility of invertible convolutions in generative flows beyond 1x1 convolutions.
• Introduce emergent and periodic convolutions with preserved receptive fields.
• Provide a stable, flexible QR-based 1x1 convolution parameterization.
• Demonstrate improved density estimation and sampling efficiency on standard datasets.

Proposed method

• Introduce emerging convolutions by chaining autoregressive convolutions to match standard receptive fields.
• Propose invertible periodic convolutions computed via decoupling in the frequency domain.
• Present a QR-based parameterization for stable and flexible 1x1 invertible convolutions.
• Provide an accelerated inversion module for autoregressive convolutions to speed up sampling.
• Evaluate density performance on Galaxy, CIFAR-10, and ImageNet benchmarks.

Experimental results

Research questions

• RQ1 Do dxd emergent and periodic convolutions improve the expressiveness and likelihoods of generative normalizing flows compared to 1x1 convolutions?
• RQ2 Can invertible dxd convolutions maintain tractable Jacobians and inverses while offering larger receptive fields?
• RQ3 Does a QR-based 1x1 convolution enhance stability and performance over PLU parametrizations?

Key findings

• Periodic and emerging 3x3 convolutions outperform prior 1x1 convolutions on Galaxy density modeling.
• Emerging convolutions match or exceed Glow on CIFAR-10 and ImageNet across varying model sizes.
• Emerging convolutions significantly reduce sampling time compared to naïve autoregressive inverses and MAF baselines.
• QR 1x1 convolutions achieve comparable performance to standard and PLU parametrizations with improved stability.
• Across CIFAR-10 and ImageNet(32x32), emerging convolutions provide notable gains, especially in smaller models.