[Paper Review] Learning Feature Hierarchies with Centered Deep Boltzmann Machines

Grégoire Montavon, Klaus‐Robert Müller|arXiv (Cornell University)|Mar 16, 2012

Generative Adversarial Networks and Image Synthesis17 references20 citations

TL;DR

This paper proposes the centered deep Boltzmann machine (C-DBM), a modified training algorithm that reparameterizes the energy function using centered states to improve optimization stability. By centering the sigmoid outputs via offset parameters, the method achieves better Hessian conditioning, enabling successful joint training of deep layers without greedy pretraining, resulting in faster convergence, improved generative modeling, and richer hierarchical feature representations on MNIST data.

ABSTRACT

Deep Boltzmann machines are in principle powerful models for extracting the hierarchical structure of data. Unfortunately, attempts to train layers jointly (without greedy layer-wise pretraining) have been largely unsuccessful. We propose a modification of the learning algorithm that initially recenters the output of the activation functions to zero. This modification leads to a better conditioned Hessian and thus makes learning easier. We test the algorithm on real data and demonstrate that our suggestion, the centered deep Boltzmann machine, learns a hierarchy of increasingly abstract representations and a better generative model of data.

Motivation & Objective

• To address the instability and poor convergence of joint training in deep Boltzmann machines.
• To improve the conditioning of the Hessian matrix during optimization by centering the output of sigmoid activation functions.
• To enable effective learning of hierarchical representations in deep generative models without greedy layer-wise pretraining.
• To evaluate whether centering leads to better discriminative features and improved generative modeling capabilities on real data.
• To investigate the impact of parameter centering on representation quality and training dynamics in deep Boltzmann machines.

Proposed method

• Reparameterize the energy function using centered states ξ = x − β, where β is an offset parameter per unit.
• Rewrite the model log-likelihood gradient using centered states to improve Hessian conditioning.
• Set initial offsets β = sigm(b₀) to enforce initial centering of sigmoid outputs.
• Maintain invariance of the Gibbs distribution under reparameterization, ensuring model equivalence.
• Implement the centered DBM by modifying weight and bias updates to use centered states in gradient computation.
• Use linear backprojection to visualize filters learned in the first and second layers of the DBM.

Experimental results

Research questions

• RQ1 Does centering the sigmoid outputs in deep Boltzmann machines lead to better Hessian conditioning and improved optimization stability?
• RQ2 Can a centered DBM successfully learn hierarchical representations without greedy layer-wise pretraining?
• RQ3 How does centering affect the diversity and quality of learned filters in the first and second layers of the DBM?
• RQ4 To what extent does centering improve the discriminative power of the top-layer representation?
• RQ5 How does centering influence the generative quality and class balance in the generated samples?

Key findings

• The centered DBM achieves faster and more stable training compared to the non-centered counterpart, with reduced training divergence.
• Centering leads to a better-conditioned Hessian, which improves the convergence behavior of the optimization process.
• The top-layer representation in the centered DBM forms distinct, label-correlated clusters after 100 epochs, indicating effective feature learning.
• Non-centered DBMs tend to collapse the top-layer representation into a low-dimensional manifold, discarding useful discriminative features.
• Second-layer filters in the centered DBM show significantly higher diversity compared to the non-centered version, indicating richer feature abstraction.
• The centered DBM generates more balanced and realistic samples, avoiding class imbalance issues seen in non-centered models.