[Paper Review] Deep Adversarial Belief Networks

Yuming Huang, Ashkan Panahi+3 more|arXiv (Cornell University)|Sep 13, 2019

Generative Adversarial Networks and Image Synthesis33 references4 citations

TL;DR

This paper introduces a novel adversarial training framework for Deep Belief Networks (DBNs) that replaces the GAN generator with a DBN and enables backpropagation-free, parallelizable training. The method achieves superior scalability and generalization, especially with limited data, outperforming standard CNNs in classification tasks when pre-trained with DBNs, while enabling flexible integration of regularization units like pooling and normalization.

ABSTRACT

We present a novel adversarial framework for training deep belief networks (DBNs), which includes replacing the generator network in the methodology of generative adversarial networks (GANs) with a DBN and developing a highly parallelizable numerical algorithm for training the resulting architecture in a stochastic manner. Unlike the existing techniques, this framework can be applied to the most general form of DBNs with no requirement for back propagation. As such, it lays a new foundation for developing DBNs on a par with GANs with various regularization units, such as pooling and normalization. Foregoing back-propagation, our framework also exhibits superior scalability as compared to other DBN and GAN learning techniques. We present a number of numerical experiments in computer vision as well as neurosciences to illustrate the main advantages of our approach.

Motivation & Objective

• To address the slow convergence and lack of regularization flexibility in traditional DBN training methods based on Gibbs sampling and maximum likelihood.
• To enable the use of advanced regularization units—such as pooling and normalization—commonly used in DNNs—within DBNs by replacing backpropagation with adversarial training.
• To improve the scalability and numerical efficiency of DBN training through a parallelizable, stochastic optimization framework inspired by GANs.
• To demonstrate the effectiveness of DBNs in modeling complex, sparse data such as biological neural spike trains and low-data vision tasks.
• To establish a new training paradigm for DBNs that matches the performance and flexibility of GANs while preserving their Bayesian, generative, and reversible properties.

Proposed method

• Replaces the generator in a GAN with a Deep Belief Network (DBN), forming a Deep Adversarial Belief Network (DABN).
• Employs an adversarial training objective where the discriminator distinguishes real data from DBN-generated samples, enabling end-to-end optimization without backpropagation through the DBN.
• Utilizes a stochastic, layer-wise training algorithm that is highly parallelizable across layers, improving scalability over conventional DBN training.
• Applies the adversarial framework to various GAN variants, including conditional GANs and WGANs, to ensure compatibility with modern optimization schemes.
• Employs a symmetric, reversible inference mechanism in the DBN to allow both generation and inference, enabling applications like feature extraction and denoising.
• Integrates standard DNN regularization techniques—such as pooling and normalization—into the DBN architecture by leveraging the adversarial training framework.

Experimental results

Research questions

• RQ1 Can adversarial training be effectively applied to DBNs to replace backpropagation and improve training efficiency?
• RQ2 Can the proposed framework support regularization units like pooling and normalization in DBNs, as in DNNs?
• RQ3 Does the adversarial training of DBNs lead to better generalization, especially with limited training data, compared to standard DBN or CNN training?
• RQ4 Can the DBN-based generative model effectively capture the statistical properties of sparse biological neural spike data?
• RQ5 How does the performance of a CNN fine-tuned with DBN-pretrained weights compare to a standard CNN when training data is scarce?

Key findings

• When trained on a small MNIST dataset (100 samples), a CNN pre-trained with a DBN achieved 77.772% accuracy with low variance (1.501%), significantly outperforming a standard CNN that achieved 70.037% with high variance (6.32%).
• With 1,000 training samples, the DBN-pretrained CNN reached 96.081% accuracy (0.622% std), compared to 94.625% for the standard CNN (0.355% std), showing improved consistency and generalization.
• The log-likelihood of real neural spike data on the DBN increased during training, confirming effective modeling of the underlying probability distribution.
• The DBN-generated firing probabilities closely matched the actual firing rates in mouse visual cortex neurons, even at low overall spike rates (~1 spike per 100 frames).
• The framework successfully enabled training of complex DBNs with convolutional and dense layers, including 4-layer convolutional DBNs and 4-layer dense DBNs, without backpropagation.
• The method demonstrated compatibility with multiple GAN variants, including conditional GANs and WGANs, and supported various optimizers and training schemes.