[Paper Review] PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications

Tim Salimans, Andrej Karpathy+2 more|arXiv (Cornell University)|Jan 19, 2017

Brain Tumor Detection and Classification567 citations

TL;DR

PixelCNN++ introduces a discretized logistic mixture likelihood and additional architectural modifications (whole-pixel conditioning, downsampling with skip connections, dropout) to improve PixelCNN performance on CIFAR-10, achieving state-of-the-art log-likelihood.

ABSTRACT

PixelCNNs are a recently proposed class of powerful generative models with tractable likelihood. Here we discuss our implementation of PixelCNNs which we make available at https://github.com/openai/pixel-cnn. Our implementation contains a number of modifications to the original model that both simplify its structure and improve its performance. 1) We use a discretized logistic mixture likelihood on the pixels, rather than a 256-way softmax, which we find to speed up training. 2) We condition on whole pixels, rather than R/G/B sub-pixels, simplifying the model structure. 3) We use downsampling to efficiently capture structure at multiple resolutions. 4) We introduce additional short-cut connections to further speed up optimization. 5) We regularize the model using dropout. Finally, we present state-of-the-art log likelihood results on CIFAR-10 to demonstrate the usefulness of these modifications.

Motivation & Objective

• Motivate improvements to the PixelCNN family for better tractable likelihood and perceptual quality in images.
• Simplify model structure while speeding training and improving convergence.
• Explore multi-resolution processing and regularization techniques to boost performance.
• Demonstrate state-of-the-art log-likelihood on CIFAR-10 with the proposed changes.

Proposed method

• Use a discretized mixture of logistic distributions for pixel likelihood instead of a 256-way softmax.
• Condition on whole pixels (R,G,B together) and model channel dependencies linearly on preceding channels.
• Incorporate downsampling with stride-2 convolutions to capture multi-resolution structure.
• Add long-range shortcut connections to recover information lost by downsampling/upsampling.
• Apply dropout regularization to reduce overfitting and improve generative quality.

Experimental results

Research questions

• RQ1 How does discretized logistic mixture likelihood compare to softmax in training speed and log-likelihood performance?
• RQ2 What is the impact of conditioning on whole pixels versus sub-pixels on model capacity and sample quality?
• RQ3 Do downsampling and shortcut connections provide comparable benefits to dilated convolutions for multi-resolution modeling?
• RQ4 What is the effect of dropout on training stability and generated image quality?
• RQ5 What are the state-of-the-art log-likelihood results on CIFAR-10 with PixelCNN++ and its variants?

Key findings

• PixelCNN++ achieves 2.92 bits per sub-pixel on CIFAR-10, improving over prior PixelCNN variants.
• Class-conditioned PixelCNN++ attains 2.94 bits per sub-pixel, with qualitative class-distinct samples.
• Softmax-based ablation trains more slowly and is less efficient than discretized logistic mixtures in this setting.
• Continuous mixture dequantization yields a variational lower bound of 3.11 bits per dimension, worse than discretized likelihood.
• Removing short-cut connections prevents training progression, highlighting their importance with downsampling.
• No dropout leads to overfitting and poorer perceptual image quality, despite high training likelihood.