[Paper Review] How to Center Binary Deep Boltzmann Machines

Jan Melchior, Asja Fischer+1 more|arXiv (Cornell University)|Nov 6, 2013

Generative Adversarial Networks and Image Synthesis17 references3 citations

TL;DR

This paper introduces a centering technique for binary Deep Boltzmann Machines (DBMs) and Restricted Boltzmann Machines (RBMs) by subtracting mean values from visible and hidden units, which stabilizes training, improves log-likelihood performance, and aligns the update direction with the natural gradient. The method is shown to eliminate the need for greedy pre-training and outperforms standard RBMs/DBMs and enhanced gradient methods in generative modeling, especially when using exponentially weighted moving averages for offset estimation.

ABSTRACT

This work analyzes centered binary Restricted Boltzmann Machines (RBMs) and binary Deep Boltzmann Machines (DBMs), where centering is done by subtracting offset values from visible and hidden variables. We show analytically that (i) centering results in a different but equivalent parameterization for artificial neural networks in general, (ii) the expected performance of centered binary RBMs/DBMs is invariant under simultaneous flip of data and offsets, for any offset value in the range of zero to one, (iii) centering can be reformulated as a different update rule for normal binary RBMs/DBMs, and (iv) using the enhanced gradient is equivalent to setting the offset values to the average over model and data mean. Furthermore, numerical simulations suggest that (i) optimal generative performance is achieved by subtracting mean values from visible as well as hidden variables, (ii) centered RBMs/DBMs reach significantly higher log-likelihood values than normal binary RBMs/DBMs, (iii) centering variants whose offsets depend on the model mean, like the enhanced gradient, suffer from severe divergence problems, (iv) learning is stabilized if an exponentially moving average over the batch means is used for the offset values instead of the current batch mean, which also prevents the enhanced gradient from diverging, (v) centered RBMs/DBMs reach higher LL values than normal RBMs/DBMs while having a smaller norm of the weight matrix, (vi) centering leads to an update direction that is closer to the natural gradient and that the natural gradient is extremly efficient for training RBMs, (vii) centering dispense the need for greedy layer-wise pre-training of DBMs, (viii) furthermore we show that pre-training often even worsen the results independently whether centering is used or not, and (ix) centering is also beneficial for auto encoders.

Motivation & Objective

• To address the lack of invariance in RBM/DBM training under data bit-flip transformations.
• To investigate whether centering visible and hidden units improves training stability and model performance.
• To analyze the relationship between centering and the natural gradient in binary RBMs and DBMs.
• To evaluate whether centering can replace greedy layer-wise pre-training in deep Boltzmann machines.
• To assess the impact of centering on autoencoders and generative modeling performance.

Proposed method

• Centering is implemented by subtracting offset values (mean of visible and hidden units) from the visible and hidden variables in RBMs and DBMs.
• The method reformulates the gradient update rule as an equivalent parameterization that maintains model equivalence while improving optimization properties.
• The authors derive analytical equivalence between centering and the enhanced gradient method when offsets are set to the average of data and model means.
• An exponentially moving average over batch means is used to estimate offset values, preventing divergence in learning.
• Theoretical analysis shows that centering leads to an update direction closer to the natural gradient, which is optimal for manifold-based optimization.
• Numerical simulations compare centered models against standard RBMs/DBMs and enhanced gradient variants under various training protocols.

Experimental results

Research questions

• RQ1 Does centering visible and hidden units in binary RBMs and DBMs improve training stability and log-likelihood performance?
• RQ2 Is the update direction in centered RBMs/DBMs closer to the natural gradient than in standard models?
• RQ3 Can centering eliminate the need for greedy layer-wise pre-training in DBMs?
• RQ4 How does centering compare to the enhanced gradient method in terms of convergence and generative performance?
• RQ5 What is the optimal strategy for estimating offset values in centering—batch mean, moving average, or model mean?

Key findings

• Centered RBMs and DBMs achieve significantly higher log-likelihood values than standard RBMs and DBMs, even with smaller weight matrix norms.
• Centering stabilizes learning and prevents divergence, especially when using an exponentially moving average for offset estimation instead of batch means.
• The enhanced gradient method is equivalent to setting offsets to the average of data and model means, but this choice leads to severe divergence without proper offset estimation.
• Centered models reach higher log-likelihood values than standard models while being less sensitive to initial learning rate and data representation.
• Centering eliminates the need for greedy pre-training in DBMs, and pre-training is shown to degrade performance regardless of centering.
• Centering also improves performance in autoencoders, indicating broader applicability beyond generative models.