[Paper Review] Guided Image Generation with Conditional Invertible Neural Networks

Lynton Ardizzone, Carsten Lüth+3 more|arXiv (Cornell University)|Jul 4, 2019

Generative Adversarial Networks and Image Synthesis42 references262 citations

TL;DR

The paper introduces conditional invertible neural networks (cINN) that fuse an invertible flow with a conditioning network to enable diverse, high-quality conditional image generation trained by maximum likelihood, demonstrated on MNIST generation and ImageNet colorization, with bidirectional latent-space manipulation.

ABSTRACT

In this work, we address the task of natural image generation guided by a conditioning input. We introduce a new architecture called conditional invertible neural network (cINN). The cINN combines the purely generative INN model with an unconstrained feed-forward network, which efficiently preprocesses the conditioning input into useful features. All parameters of the cINN are jointly optimized with a stable, maximum likelihood-based training procedure. By construction, the cINN does not experience mode collapse and generates diverse samples, in contrast to e.g. cGANs. At the same time our model produces sharp images since no reconstruction loss is required, in contrast to e.g. VAEs. We demonstrate these properties for the tasks of MNIST digit generation and image colorization. Furthermore, we take advantage of our bi-directional cINN architecture to explore and manipulate emergent properties of the latent space, such as changing the image style in an intuitive way.

Motivation & Objective

• Introduce a new architecture: conditional invertible neural network (cINN) that combines an INN with a conditioning pathway.
• Provide a stable maximum likelihood-based training procedure for jointly optimizing INN and conditioning components.
• Demonstrate diversity and image sharpness without reconstruction losses, addressing mode collapse in conditional generation.
• Explore emergent latent-space properties and enable intuitive manipulation of image attributes via bidirectional design.

Proposed method

• Extend affine coupling blocks to conditional inputs by appending conditioning c to the subnetworks.
• Use a conditioning network h to preprocess c into a representation c~ and plug into the likelihood training objective.
• Train with maximum likelihood using pX(x; c, θ) = pZ(f(x; c, θ)) |det(df/dx)|, minimizing the loss E[-log pX] + regularization.
• Sample new images by drawing z ~ pZ(z) and applying the inverse f^{-1}(z; c, θML).
• Avoid mode collapse since latent vectors outside pZ incur large likelihood loss (ML training).
• Incorporate data augmentation (noise) and stability tricks (soft-clamping of s, Xavier initialization, soft channel permutations).

Experimental results

Research questions

• RQ1 Can a conditional invertible architecture generate diverse, high-quality images conditioned on arbitrary inputs without mode collapse?
• RQ2 Does maximum likelihood training suffice to train cINNs for conditional generation, avoiding reconstruction losses and producing sharp results?
• RQ3 How does a conditioning network improve the conditioning signal and overall generation quality for complex conditioning like images?
• RQ4 What latent-space manipulations become possible with a bidirectional cINN when conditioning on inputs like grayscale or style information?

Key findings

• The cINN framework produces diverse, sharp conditional images without relying on reconstruction losses.
• Maximum likelihood training yields stable optimization and mitigates mode collapse compared to some GAN-based approaches.
• The method demonstrates conditional MNIST digit generation and diverse colorization of ImageNet, illustrating effective conditioning and diversity.
• The bidirectional design enables manipulation of emergent latent-space properties, such as changing image style through latent variables.