[Paper Review] Joint Training Deep Boltzmann Machines for Classification

Ian Goodfellow, Aaron Courville+1 more|arXiv (Cornell University)|Jan 16, 2013

Generative Adversarial Networks and Image Synthesis8 references13 citations

TL;DR

This paper introduces multi-prediction (MP) training for joint learning in deep Boltzmann machines (DBMs), enabling end-to-end training without greedy pretraining. The method uses a novel variational approximation and the multi-inference trick to achieve state-of-the-art classification accuracy (0.91% test error) with a single, unified model, outperforming prior approaches that required separate fine-tuning with MLPs.

ABSTRACT

We introduce a new method for training deep Boltzmann machines jointly. Prior methods of training DBMs require an initial learning pass that trains the model greedily, one layer at a time, or do not perform well on classification tasks. In our approach, we train all layers of the DBM simultaneously, using a novel training procedure called multi-prediction training. The resulting model can either be interpreted as a single generative model trained to maximize a variational approximation to the generalized pseudolikelihood, or as a family of recurrent networks that share parameters and may be approximately averaged together using a novel technique we call the multi-inference trick. We show that our approach performs competitively for classification and outperforms previous methods in terms of accuracy of approximate inference and classification with missing inputs.

Motivation & Objective

• Address the limitation of prior DBM training methods that require greedy, layerwise pretraining followed by fine-tuning with a separate MLP for classification.
• Develop a unified training procedure that jointly optimizes all DBM components, including the class label, in a single stage.
• Enable the DBM to perform well not only in standard classification but also in challenging scenarios like missing inputs and general probabilistic queries.
• Demonstrate that a single, generatively trained DBM can match or exceed the performance of models requiring task-specific architectural extensions.

Proposed method

• Introduce multi-prediction (MP) training, a novel training procedure that maximizes a variational approximation to the generalized pseudolikelihood for joint training of all DBM layers.
• Formulate the DBM as a family of recurrent networks with shared parameters, enabling approximate averaging via the multi-inference trick to improve inference stability.
• Use block Gibbs sampling and mean field inference with fixed-point updates to efficiently estimate gradients, leveraging conditional independence across layers.
• Apply Rao-Blackwellization and a specialized negative phase with mean field components to reduce variance in the gradient estimate during PCD learning.
• Train the full DBM in one stage using a single objective function, eliminating the need for greedy pretraining or separate fine-tuning networks.
• Leverage the multi-inference trick to combine multiple inference paths, enhancing robustness and accuracy in approximate inference, especially under missing data.

Experimental results

Research questions

• RQ1 Can a deep Boltzmann machine be jointly trained across all layers without greedy pretraining while maintaining strong classification performance?
• RQ2 Does multi-prediction training enable better approximate inference and classification accuracy compared to standard DBM training with or without centering?
• RQ3 Can a single, generatively trained DBM handle missing inputs and general queries as effectively as models with task-specific fine-tuning?
• RQ4 How does the performance of the MP-DBM compare to the state-of-the-art DBM method (Salakhutdinov & Hinton, 2009) in terms of test error and robustness to hyperparameter settings?
• RQ5 To what extent does the multi-inference trick improve inference quality and generalization in probabilistic DBM inference?

Key findings

• The MP-DBM achieves a test error of 0.91% on MNIST by jointly training all layers in a single stage, outperforming the prior state-of-the-art method (0.95%) that required an additional MLP for fine-tuning.
• MP training is significantly less sensitive to hyperparameter choices than centering-based methods, with consistent performance across a wide range of learning rates and momentum schedules.
• When classifying with missing inputs, the MP-DBM outperforms both standard DBMs and centering-based DBMs for most levels of input corruption.
• In general probabilistic queries involving multiple missing variables, the MP-DBM demonstrates superior performance compared to standard and centered DBMs, especially for larger query sets.
• The best-performing MP-DBM with twice the number of hidden units and the multi-inference trick achieves 0.91% test error, matching or exceeding the performance of models that use a separate MLP head.
• The method enables a single, unified model to serve as both a powerful generative model and an accurate classifier, eliminating the need for task-specific architectural extensions.