[Paper Review] A Deep and Tractable Density Estimator

Benigno Uría, Iain Murray+1 more|arXiv (Cornell University)|Oct 7, 2013

Gaussian Processes and Bayesian Inference19 references104 citations

TL;DR

This paper introduces an order-agnostic training procedure for Neural Autoregressive Density Estimators (NADE) that simultaneously trains a NADE model for every possible variable ordering by sharing parameters across all orderings. The method enables exact marginalization and conditioning, supports deep architectures with linear computational overhead, and allows on-the-fly ensemble creation, achieving state-of-the-art density estimation performance, including on the BSDS300 image patch dataset with a 6-layer model scoring 155.2 log-likelihood.

ABSTRACT

The Neural Autoregressive Distribution Estimator (NADE) and its real-valued version RNADE are competitive density models of multidimensional data across a variety of domains. These models use a fixed, arbitrary ordering of the data dimensions. One can easily condition on variables at the beginning of the ordering, and marginalize out variables at the end of the ordering, however other inference tasks require approximate inference. In this work we introduce an efficient procedure to simultaneously train a NADE model for each possible ordering of the variables, by sharing parameters across all these models. We can thus use the most convenient model for each inference task at hand, and ensembles of such models with different orderings are immediately available. Moreover, unlike the original NADE, our training procedure scales to deep models. Empirically, ensembles of Deep NADE models obtain state of the art density estimation performance.

Motivation & Objective

• To address the limitation of fixed variable ordering in NADE, which hinders efficient marginalization and conditioning on arbitrary subsets of variables.
• To enable training of deep NADE models with only linear computational overhead relative to shallow models.
• To allow on-the-fly construction of ensembles of NADE models across different orderings without additional training cost.
• To improve density estimation performance on real-world datasets, particularly for high-dimensional data like image patches.

Proposed method

• Train a single shared model that implicitly represents a NADE for every possible variable ordering through parameter tying across all permutations.
• Optimize the average negative log-likelihood over all orderings using stochastic gradient descent, enabling joint training across factorial number of models.
• Use recursive computation of hidden activations (via Equation 5) to maintain efficiency, even in deep architectures.
• Leverage the inconsistency across orderings to generate ensembles on the fly, improving generalization and likelihood estimates.
• Apply mixture density networks (MDNs) for real-valued data in RNADE extensions, enabling modeling of continuous distributions.
• Use early stopping and learning rate scheduling to prevent overfitting, even with deep models and large numbers of layers.

Experimental results

Research questions

• RQ1 Can we train a single model that supports exact marginalization and conditioning for any subset of variables without fixing a single ordering?
• RQ2 Is it possible to scale NADE to deep architectures without incurring super-linear computational costs?
• RQ3 Can ensembles of NADE models trained across different orderings improve density estimation performance without explicit training of the ensemble?
• RQ4 Does the order-agnostic training procedure yield state-of-the-art performance on benchmark datasets like BSDS300 image patches?

Key findings

• The proposed method achieved a test-set log-likelihood of 157.0 on 8×8 natural image patches from BSDS300, surpassing all prior methods including mixtures of Gaussians and fixed-order RNADEs.
• A 6-hidden-layer RNADE trained with the order-agnostic procedure achieved 155.2 log-likelihood, outperforming the fixed-order baseline (152.1) and all shallower models.
• Ensembles of 32 orderings (EoRNADE 6hl 32 ord.) achieved 157.0 log-likelihood, demonstrating that on-the-fly ensembling improves performance with minimal inference cost.
• The method scales linearly with depth: training a 6-layer model incurred only a linear increase in computation relative to shallow models.
• No overfitting was observed even with 6 hidden layers, and validation cost did not increase steadily, suggesting potential for further performance gains with deeper models.
• The approach enables exact sampling and marginalization, unlike MCMC-based or variational methods, while maintaining tractable likelihood computation.