[Paper Review] Deep AutoRegressive Networks

Karol Gregor, Ivo Danihelka+3 more|arXiv (Cornell University)|Oct 31, 2013

Generative Adversarial Networks and Image Synthesis28 references112 citations

TL;DR

This paper introduces Deep AutoRegressive Networks (DARNs), a deep generative autoencoder with autoregressive stochastic hidden layers that enable fast, exact ancestral sampling for efficient generation. The model is trained via minimum description length (MDL) optimization, which approximates variational inference and achieves state-of-the-art generative performance on MNIST, Atari games, and UCI datasets.

ABSTRACT

We introduce a deep, generative autoencoder capable of learning hierarchies of distributed representations from data. Successive deep stochastic hidden layers are equipped with autoregressive connections, which enable the model to be sampled from quickly and exactly via ancestral sampling. We derive an efficient approximate parameter estimation method based on the minimum description length (MDL) principle, which can be seen as maximising a variational lower bound on the log-likelihood, with a feedforward neural network implementing approximate inference. We demonstrate state-of-the-art generative performance on a number of classic data sets: several UCI data sets, MNIST and Atari 2600 games.

Motivation & Objective

• To develop a deep generative autoencoder that enables fast, exact sampling through ancestral sampling, overcoming slow, correlated sampling in prior models.
• To provide a theoretically grounded training method using the minimum description length (MDL) principle, ensuring compact and non-redundant representations.
• To integrate autoregressive connections within stochastic hidden layers, capturing intra-layer dependencies efficiently without prohibitive computational cost.
• To enable scalable, deep architectures with alternating stochastic and deterministic layers, supporting hierarchical representation learning.
• To demonstrate state-of-the-art generative performance across diverse data modalities, including images and sequential data.

Proposed method

• The model uses a deep architecture with stochastic hidden layers connected via autoregressive dependencies, where each unit depends on previous units in the same layer and the previous layer.
• The decoder uses ancestral sampling: starting from the deepest layer, units are sampled top-down, one at a time, to generate exact samples without Markov chain burn-in.
• The encoder performs bottom-up, left-to-right inference to approximate the posterior distribution over hidden representations given an observation.
• Training is performed by minimizing the MDL cost, which corresponds to minimizing the Helmholtz variational free energy, using stochastic gradient descent.
• Backpropagation through stochastic units is enabled via a reparameterization trick with a control variate baseline to reduce gradient variance.
• The baseline is a first-order Taylor approximation of the network output, evaluated at h_i = 0.5, to improve gradient estimation stability.

Experimental results

Research questions

• RQ1 Can autoregressive connections within stochastic hidden layers enable fast, exact ancestral sampling in deep generative models?
• RQ2 Does training via the minimum description length (MDL) principle yield better generative performance and more compact representations than standard autoencoder regularization?
• RQ3 Can deep architectures with alternating stochastic and deterministic layers be effectively trained and scaled to complex data such as images and video frames?
• RQ4 How does the inclusion of intra-layer autoregressive dependencies compare to undirected or fully connected lateral connections in terms of computational efficiency and modeling capacity?
• RQ5 To what extent can DARNs achieve state-of-the-art generative performance on benchmark datasets like MNIST and Atari 2600 games?

Key findings

• DARNs achieve state-of-the-art negative log-likelihood on MNIST, with a test set score of 108.5 bits/dim, outperforming previous models.
• On Atari 2600 games, DARNs achieve test set negative log-likelihoods of 19.9 (Freeway), 23.7 (Pong), 113.0 (Space Invaders), 139.4 (River Raid), and 217.9 (Sea Quest).
• The model generates high-quality, diverse samples that include novel combinations of objects not seen in training, as shown in samples from locally connected DARNs.
• The use of a control variate baseline significantly reduces gradient variance during training, enabling stable optimization through stochastic units.
• The model scales effectively to convolutional and locally connected architectures, maintaining high sample quality and training efficiency.
• The MDL-based training objective leads to compact, non-redundant representations that are both predictive and generative.