[Paper Review] Restricted Boltzmann Machine and Deep Belief Network: Tutorial and Survey

Benyamin Ghojogh, Ali Ghodsi+2 more|arXiv (Cornell University)|Jul 26, 2021

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This tutorial and survey paper provides a comprehensive overview of Restricted Boltzmann Machines (RBMs) and Deep Belief Networks (DBNs), explaining their theoretical foundations in statistical physics and probabilistic graphical models, detailing training via contrastive divergence and Gibbs sampling, and demonstrating their use in deep learning through greedy pre-training and fine-tuning. The key contribution is a unified, accessible guide to RBM and DBN architectures, highlighting their role in overcoming vanishing gradients and enabling deep generative models prior to modern deep learning breakthroughs.

ABSTRACT

This is a tutorial and survey paper on Boltzmann Machine (BM), Restricted Boltzmann Machine (RBM), and Deep Belief Network (DBN). We start with the required background on probabilistic graphical models, Markov random field, Gibbs sampling, statistical physics, Ising model, and the Hopfield network. Then, we introduce the structures of BM and RBM. The conditional distributions of visible and hidden variables, Gibbs sampling in RBM for generating variables, training BM and RBM by maximum likelihood estimation, and contrastive divergence are explained. Then, we discuss different possible discrete and continuous distributions for the variables. We introduce conditional RBM and how it is trained. Finally, we explain deep belief network as a stack of RBM models. This paper on Boltzmann machines can be useful in various fields including data science, statistics, neural computation, and statistical physics.

Motivation & Objective

• To provide a comprehensive tutorial and survey of Boltzmann Machines (BMs), Restricted Boltzmann Machines (RBMs), and Deep Belief Networks (DBNs) for researchers in machine learning and data science.
• To bridge foundational concepts from statistical physics, Ising model, and Hopfield networks to modern RBM and DBN architectures.
• To explain the training mechanisms of RBMs using maximum likelihood estimation, contrastive divergence, and Gibbs sampling for generative modeling.
• To demonstrate how DBNs enable deep learning through greedy layer-wise pre-training and subsequent fine-tuning with backpropagation.
• To highlight the historical significance of RBMs and DBNs in overcoming the vanishing gradient problem and enabling the resurgence of deep neural networks.

Proposed method

• Derives the energy-based model formulation of BM and RBM using the Boltzmann/Gibbs distribution and statistical physics principles.
• Explains conditional distributions of visible and hidden units in RBMs, enabling joint probability modeling.
• Details Gibbs sampling for generating visible and hidden variables in RBMs, forming the basis for model sampling and training.
• Introduces contrastive divergence as an efficient approximation for maximum likelihood training of RBMs, reducing computational cost.
• Presents the greedy layer-wise pre-training algorithm for DBNs, where each layer is trained as an RBM before fine-tuning with backpropagation.
• Describes conditional RBMs (CRBMs) and their training for structured prediction tasks, extending RBM capabilities.

Experimental results

Research questions

• RQ1 How do RBMs and DBNs build upon the Ising model and Hopfield network in the context of energy-based learning?
• RQ2 What is the role of contrastive divergence in enabling efficient training of RBMs via approximate maximum likelihood estimation?
• RQ3 How does the greedy pre-training of DBNs mitigate the vanishing gradient problem in deep networks?
• RQ4 In what ways can RBMs and DBNs be adapted for structured prediction and dimensionality reduction through autoencoder configurations?
• RQ5 What are the key theoretical and practical differences between RBMs, DBNs, and other energy-based models like the Helmholtz machine or Deep Boltzmann Machines?

Key findings

• RBMs and DBNs were instrumental in overcoming the vanishing gradient problem in deep networks by enabling effective weight initialization through greedy pre-training.
• Contrastive divergence enables efficient, approximate maximum likelihood training of RBMs by minimizing the difference between positive and negative phase data distributions.
• DBNs can be used as deep autoencoders for dimensionality reduction, learning low-dimensional representations of data through symmetric encoder-decoder architectures.
• The success of DBNs in applications such as speech recognition and action recognition demonstrates their effectiveness in modeling complex, high-dimensional data before the rise of modern deep learning frameworks.
• Despite the later dominance of ReLU and dropout in modern deep learning, RBMs and DBNs laid the theoretical and practical groundwork for the resurgence of deep neural networks after 2006.