[Paper Review] Conditional Image Generation with PixelCNN Decoders

Aäron van den Oord, Nal Kalchbrenner+4 more|arXiv (Cornell University)|Jun 16, 2016

Generative Adversarial Networks and Image Synthesis32 references799 citations

TL;DR

This paper introduces Gated PixelCNN and Conditional PixelCNN to model and generate images conditioned on labels or embeddings, achieving state-of-the-art likelihoods with faster training than PixelRNN and enabling conditional image synthesis and autoencoder decoding.

ABSTRACT

This work explores conditional image generation with a new image density model based on the PixelCNN architecture. The model can be conditioned on any vector, including descriptive labels or tags, or latent embeddings created by other networks. When conditioned on class labels from the ImageNet database, the model is able to generate diverse, realistic scenes representing distinct animals, objects, landscapes and structures. When conditioned on an embedding produced by a convolutional network given a single image of an unseen face, it generates a variety of new portraits of the same person with different facial expressions, poses and lighting conditions. We also show that conditional PixelCNN can serve as a powerful decoder in an image autoencoder. Additionally, the gated convolutional layers in the proposed model improve the log-likelihood of PixelCNN to match the state-of-the-art performance of PixelRNN on ImageNet, with greatly reduced computational cost.

Motivation & Objective

• Motivate conditional image modeling for tasks like denoising, inpainting, and conditional generation of diverse scenes.
• Develop a faster, effective autoregressive image model by upgrading PixelCNN to a gated variant and address receptive-field blind spots.
• Demonstrate conditioning on class labels and embeddings to enable diverse, realistic samples across multiple datasets.
• Explore using Conditional PixelCNN as an image decoder in autoencoders to learn high-level representations.

Proposed method

• Introduce Gated PixelCNN with a gating mechanism to replace standard activations.
• Combine two convolutional stacks (horizontal and vertical) to eliminate receptive-field blind spots.
• Develop Conditional PixelCNN by injecting conditioning vectors into layer activations (and optionally spatial maps) to model p(x|h).
• Formulate a PixelCNN auto-encoder by replacing the decoder with a Conditional PixelCNN and training end-to-end.

Experimental results

Research questions

• RQ1 Can a gated, autoregressive CNN match PixelRNN performance while reducing training time?
• RQ2 Does conditioning PixelCNN on class labels or embeddings produce diverse, high-quality samples across tasks?
• RQ3 Can Conditional PixelCNN serve effectively as a decoder in autoencoders to learn different latent representations?
• RQ4 How does conditioning influence log-likelihood and visual diversity on CIFAR-10 and ImageNet-scale datasets?

Key findings

• Gated PixelCNN achieves comparable log-likelihood to PixelRNN on CIFAR-10 and ImageNet while using less than half the training time.
• Class-conditioned sampling yields clearly distinct and diverse samples across 8 classes, with variations in pose and background.
• Portrait embeddings conditioned sampling generates new faces of the same person with varied expressions, poses, and lighting; interpolation in embedding space yields smooth transitions.
• PixelCNN auto-encoder reconstructions show qualitative differences, suggesting the encoder captures higher-level structure when used with a probabilistic PixelCNN decoder.
• On ImageNet variants, Gated PixelCNN outperforms PixelRNN in negative log-likelihood for 32x32 and 64x64 settings, approaching state-of-the-art while remaining efficient.
• The two-stack (horizontal and vertical) architecture removes the blind spot in receptive fields, enabling better modeling of pixel dependencies.