[Paper Review] Deep Generative Stochastic Networks Trainable by Backprop

Yoshua Bengio, Eric Laufer+2 more|arXiv (Cornell University)|Jun 5, 2013

Generative Adversarial Networks and Image SynthesisComputer Science39 references239 citations

TL;DR

This paper introduces Generative Stochastic Networks (GSNs), a framework that trains deep generative models by learning the transition operator of a Markov chain instead of directly modeling the data distribution. By using denoising autoencoder-style training with backpropagation, GSNs simplify density estimation by approximating unimodal conditional distributions, enabling efficient training without intractable partition functions and achieving state-of-the-art log-likelihood bounds on MNIST and TFD datasets.

ABSTRACT

We introduce a novel training principle for probabilistic models that is an alternative to maximum likelihood. The proposed Generative Stochastic Networks (GSN) framework is based on learning the transition operator of a Markov chain whose stationary distribution estimates the data distribution. The transition distribution of the Markov chain is conditional on the previous state, generally involving a small move, so this conditional distribution has fewer dominant modes, being unimodal in the limit of small moves. Thus, it is easier to learn because it is easier to approximate its partition function, more like learning to perform supervised function approximation, with gradients that can be obtained by backprop. We provide theorems that generalize recent work on the probabilistic interpretation of denoising autoencoders and obtain along the way an interesting justification for dependency networks and generalized pseudolikelihood, along with a definition of an appropriate joint distribution and sampling mechanism even when the conditionals are not consistent. GSNs can be used with missing inputs and can be used to sample subsets of variables given the rest. We validate these theoretical results with experiments on two image datasets using an architecture that mimics the Deep Boltzmann Machine Gibbs sampler but allows training to proceed with simple backprop, without the need for layerwise pretraining.

Motivation & Objective

• To address the challenge of training deep unsupervised models due to intractable partition functions in probabilistic models.
• To develop a training framework that transforms unsupervised density estimation into a supervised-like function approximation problem.
• To enable end-to-end training of deep generative models using standard backpropagation, avoiding the need for layerwise pretraining or MCMC sampling.
• To improve model mixing and representation learning by incorporating latent variables into the Markov chain state.
• To validate the framework on image datasets with performance comparable to or better than existing models like RBMs and DBMs.

Proposed method

• The GSN framework models the data distribution by learning the transition operator of a Markov chain, where each step involves sampling from a conditional distribution $ P_{\theta}(X|\tilde{X}) $, with $ \tilde{X} $ being a corrupted version of $ X $.
• The method uses a denoising autoencoder-like mechanism: input $ X $ is corrupted to $ \tilde{X} $, and the model learns to reconstruct $ X $ from $ \tilde{X} $, effectively learning a conditional distribution that is simpler (unimodal) than the true data distribution.
• The framework generalizes to include latent variables $ H $, allowing the Markov chain state to include both observed $ X $ and latent $ H $, enabling richer representations and easier mixing.
• Training is performed via backpropagation on the reconstruction log-likelihood objective, treating the denoising task as a supervised learning problem with gradients computed efficiently.
• The model uses a stochastic gradient descent optimization with learning rate decay and momentum, and is trained without layerwise pretraining.
• Model quality is evaluated using a Parzen density estimator on generated samples, providing a lower bound on the true log-likelihood.

Experimental results

Research questions

• RQ1 Can we train deep generative models without computing intractable partition functions by learning a Markov chain transition operator?
• RQ2 Does learning a denoising conditional distribution $ P_{\theta}(X|\tilde{X}) $, which is simpler than the true data distribution $ P(X) $, enable more effective training via backpropagation?
• RQ3 Can the inclusion of latent variables $ H $ in the Markov chain state improve representational power and mixing efficiency in deep models?
• RQ4 How does the performance of GSNs compare to existing models like RBMs, DBMs, and DBNs in terms of log-likelihood and sample quality?
• RQ5 Can GSNs be trained end-to-end with standard backpropagation and achieve fast convergence and good generalization on image datasets?

Key findings

• The two-layer GSN achieved a Parzen log-likelihood bound of 214 ± 1.1 on the MNIST test set, significantly outperforming a single-layer denoising autoencoder (−152 ± 2.2) and matching or exceeding the performance of deeper models like DBNs and DBMs.
• The 2-layer GSN outperformed a 2-hidden-layer DBN (138 ± 2) and a 3-hidden-layer DBM (32 ± 2), demonstrating superior sample quality and likelihood estimation.
• The GSN achieved a Parzen log-likelihood bound of 1890 ± 29 on the TFD dataset, comparable to a 2-hidden-layer DBN (1908 ± 66), indicating strong performance on a different image distribution.
• Samples generated by the GSN showed fast mixing and high sharpness, with visual quality improving rapidly within 25 epochs, indicating fast convergence.
• Conditional sampling experiments demonstrated that the GSN could stochastically complete missing inputs (e.g., left half of an image), generating plausible completions consistent with the trained conditional distribution.
• The framework enabled training without layerwise pretraining and without explicit MCMC sampling, relying solely on backpropagation for end-to-end optimization.