[Paper Review] Towards Conceptual Compression

Karol Gregor, Frederic Besse+3 more|arXiv (Cornell University)|Apr 29, 2016

Generative Adversarial Networks and Image SynthesisComputer Science105 citations

TL;DR

The paper introduces convolutional DRAW, a recurrent variational auto-encoder that achieves state-of-the-art image modeling, provides hierarchical latent representations that separate global concepts from details, and enables high-quality conceptual lossy compression by storing only high-level latents.

ABSTRACT

We introduce a simple recurrent variational auto-encoder architecture that significantly improves image modeling. The system represents the state-of-the-art in latent variable models for both the ImageNet and Omniglot datasets. We show that it naturally separates global conceptual information from lower level details, thus addressing one of the fundamentally desired properties of unsupervised learning. Furthermore, the possibility of restricting ourselves to storing only global information about an image allows us to achieve high quality 'conceptual compression'.

Motivation & Objective

• Demonstrate that a simple recurrent variational auto-encoder can outperform prior latent-variable models on image datasets (Omniglot, ImageNet, CIFAR-10).
• Show that multiple hierarchical stochastic layers naturally separate global concepts from low-level details.
• Propose and evaluate a notion of conceptual compression by storing only high-level latent variables and generating the rest.

Proposed method

• Propose convolutional DRAW, a recurrent VAE with convolutional LSTM components for both encoder and decoder.
• Use a multi-layer, stacked latent-variable structure where higher layers influence priors of lower layers and provide coarse guidance for generation.
• Model latent variables with Gaussian approximate posterior q and Gaussian prior p, with KL divergences driving the information flow (L^z_t).
• Train with stochastic gradient-based optimization of the variational objective and employ arithmetic coding considerations for compression.
• Investigate input-cost scaling (beta parameter) to control emphasis on high-level structure vs. fine details.

Experimental results

Research questions

• RQ1 Can a recurrent, convolutional VAE achieve state-of-the-art likelihoods on Omniglot, CIFAR-10, and ImageNet datasets?
• RQ2 Does stacking multiple stochastic layers induce a meaningful global-to-detail information hierarchy suitable for conceptual compression?
• RQ3 Can high-level latent variables alone support high-quality lossy compression, and how does beta scaling affect generated image quality?
• RQ4 How does the information distribution across layers/time steps evolve during inference (which levels capture early vs. late information)?

Key findings

• Convolutional DRAW achieves superior likelihoods to prior latent-variable models across Omniglot and ImageNet benchmarks, outperforming earlier VAE/DRAW variants.
• The model learns a natural separation of global conceptual information from low-level details, enabling conceptual compression by storing higher-level latents and generating details later.
• Higher information is concentrated early in time at the top layer, with lower layers accumulating information later, supporting a global-to-detail generation process.
• Adjusting the input-cost scale (beta) shifts focus from fine details to broader structures, producing cleaner high-level representations at lower beta values.
• At larger iteration depths, training speed can improve per data pass and final performance can surpass shallower configurations.