[Paper Review] Neural Variational Inference and Learning in Belief Networks

Andriy Mnih, Karol Gregor|arXiv (Cornell University)|Jan 31, 2014

Domain Adaptation and Few-Shot Learning27 references204 citations

TL;DR

This paper introduces Neural Variational Inference and Learning (NVIL), a method that trains directed belief networks using a feedforward inference network to enable fast, exact sampling from the variational posterior. By jointly optimizing the model and inference network via a variational lower bound with variance-reduced gradient estimation, NVIL achieves state-of-the-art performance on MNIST and Reuters RCV1, outperforming the wake-sleep algorithm and prior methods.

ABSTRACT

Highly expressive directed latent variable models, such as sigmoid belief networks, are difficult to train on large datasets because exact inference in them is intractable and none of the approximate inference methods that have been applied to them scale well. We propose a fast non-iterative approximate inference method that uses a feedforward network to implement efficient exact sampling from the variational posterior. The model and this inference network are trained jointly by maximizing a variational lower bound on the log-likelihood. Although the naive estimator of the inference model gradient is too high-variance to be useful, we make it practical by applying several straightforward model-independent variance reduction techniques. Applying our approach to training sigmoid belief networks and deep autoregressive networks, we show that it outperforms the wake-sleep algorithm on MNIST and achieves state-of-the-art results on the Reuters RCV1 document dataset.

Motivation & Objective

• To address the challenge of training highly expressive directed latent variable models, such as sigmoid belief networks, on large datasets where exact inference is intractable.
• To overcome the limitations of existing approximate inference methods that either scale poorly or require model-specific derivations.
• To develop a general, scalable, and memory-efficient training procedure for directed graphical models that supports both discrete and continuous latent variables.
• To enable joint training of the generative model and an inference network using a variational lower bound with practical gradient estimation.
• To demonstrate the effectiveness of the approach on benchmark datasets like MNIST and Reuters RCV1, achieving state-of-the-art results.

Proposed method

• Proposes a feedforward neural network as an inference network to perform fast, exact sampling from the variational posterior distribution for a given observation.
• Trains the model and inference network jointly by maximizing a variational lower bound on the log-likelihood, using stochastic gradient estimation.
• Applies general-purpose variance reduction techniques—such as baseline subtraction and control variates—to make the gradient estimator of the inference network practical despite high variance.
• Uses the REINFORCE algorithm framework to backpropagate through the stochastic sampling process, enabling end-to-end training.
• Supports both discrete and continuous latent variables and complex dependency structures in the variational posterior, unlike prior methods with restricted assumptions.
• Enables online learning by avoiding the need to store latent variable states across training examples, improving memory efficiency.

Experimental results

Research questions

• RQ1 Can a non-iterative, feedforward inference network be used to enable efficient and scalable training of directed belief networks?
• RQ2 Can variance-reduced gradient estimation make the naive REINFORCE gradient estimator practical for training the inference network?
• RQ3 Does the proposed method outperform existing algorithms like wake-sleep in training sigmoid belief networks?
• RQ4 Can NVIL achieve state-of-the-art performance on large-scale document modeling tasks with complex, high-dimensional data?
• RQ5 Is the method general enough to handle both discrete and continuous latent variables and complex posterior structures without model-specific derivations?

Key findings

• NVIL outperforms the wake-sleep algorithm on MNIST, achieving a test negative log-likelihood of 94.5 with a 200-200-200 SBN architecture.
• On the Reuters RCV1 dataset, the fDARN model with 200 latent variables achieved a perplexity of 598, setting a new state-of-the-art record.
• The fDARN model with 50 latent variables achieved a perplexity of 724 on RCV1, outperforming DocNADE’s best published result of 742.
• NVIL achieved competitive results on 20 Newsgroups, with a perplexity of 917 for fDARN (50 latent variables), outperforming LDA and Replicated Softmax.
• The method is scalable and memory-efficient, enabling online learning due to the absence of latent state storage between updates.
• The approach generalizes across model architectures and supports both discrete and continuous latent variables, unlike prior variational methods with restricted assumptions.