[Paper Review] On the mapping between Hopfield networks and Restricted Boltzmann Machines

Matthew Smart, Anton Zilman|arXiv (Cornell University)|Jan 27, 2021

Generative Adversarial Networks and Image Synthesis25 references4 citations

TL;DR

This paper presents an exact mapping from Hopfield networks (HNs) with correlated binary patterns to Restricted Boltzmann Machines (RBMs) with binary visible units and Gaussian hidden units, enabling a direct conversion of HN weights into RBM weight initializations. The method provides a significant training advantage, with Hopfield-initialized RBMs outperforming randomly initialized models even after minimal training, particularly on MNIST data.

ABSTRACT

Hopfield networks (HNs) and Restricted Boltzmann Machines (RBMs) are two important models at the interface of statistical physics, machine learning, and neuroscience. Recently, there has been interest in the relationship between HNs and RBMs, due to their similarity under the statistical mechanics formalism. An exact mapping between HNs and RBMs has been previously noted for the special case of orthogonal (uncorrelated) encoded patterns. We present here an exact mapping in the case of correlated pattern HNs, which are more broadly applicable to existing datasets. Specifically, we show that any HN with $N$ binary variables and $p<N$ arbitrary binary patterns can be transformed into an RBM with $N$ binary visible variables and $p$ gaussian hidden variables. We outline the conditions under which the reverse mapping exists, and conduct experiments on the MNIST dataset which suggest the mapping provides a useful initialization to the RBM weights. We discuss extensions, the potential importance of this correspondence for the training of RBMs, and for understanding the performance of deep architectures which utilize RBMs.

Motivation & Objective

• To establish an exact correspondence between Hopfield networks and RBMs for arbitrary (correlated) binary patterns, extending prior work limited to orthogonal patterns.
• To enable practical RBM training improvements by leveraging HN-derived weight initialization.
• To characterize the reverse mapping from RBMs back to HNs and provide a matrix-factorization approach for approximate reconstruction when exact reverse mapping is not feasible.
• To demonstrate the utility of the mapping in reducing training time and model size while maintaining or improving performance on generative and classification tasks.

Proposed method

• The paper derives an exact transformation from a Hopfield network with $N$ binary units and $p < N$ arbitrary binary patterns to an RBM with $N$ binary visible units and $p$ Gaussian hidden units.
• The mapping uses the projection rule $\mathbf{J} = \bm{\xi}(\bm{\xi}^T\bm{\xi})^{-1}\bm{\xi}^T$ to define the HN Hamiltonian, which is then re-expressed as an equivalent RBM energy function.
• The RBM is constructed such that its weight matrix corresponds to the HN's interaction matrix, with visible units representing the HN spins and hidden units representing the pattern projections.
• The reverse mapping is analyzed using matrix factorization, with conditions derived under which an HN can be reconstructed from an RBM.
• The method enables efficient computation of the HN partition function by summing out spins via the RBM representation.
• Experiments use MNIST data to validate the initialization: HN weights are orthogonalized and used to initialize RBM weights, followed by training and evaluation on generative and classification tasks.

Experimental results

Research questions

• RQ1 Can an exact mapping be established between Hopfield networks and RBMs for arbitrary (non-orthogonal) binary patterns, extending beyond the special case of uncorrelated patterns?
• RQ2 What are the conditions under which the reverse mapping from RBMs to HNs is possible, and how can it be approximated when exact reconstruction fails?
• RQ3 Does the HN-to-RBM mapping provide a superior initialization for RBM training compared to random or other heuristic initializations?
• RQ4 How does the HN-based initialization affect training efficiency and model performance on real-world datasets like MNIST?
• RQ5 Can this correspondence improve the performance of deep architectures built from RBMs, such as deep belief networks or deep Boltzmann machines?

Key findings

• The HN-to-RBM mapping enables exact conversion of any Hopfield network with $N$ binary units and $p < N$ arbitrary binary patterns into an RBM with $N$ binary visible and $p$ Gaussian hidden units.
• The reverse mapping from RBMs to HNs is possible only under specific conditions, with matrix factorization used to construct approximate HNs when exact reconstruction fails.
• On the MNIST dataset, RBMs initialized with orthogonalized HN weights achieve $3.0\%$ training error with $k=100$ hidden units after just 1 epoch, outperforming randomly initialized models after 50 epochs.
• Even with only $k=10$ hidden units, the HN-initialized PoE model performs as well as or better than a $k=100$ randomly initialized PoE model at every training epoch.
• The $k=100$ HN-initialized RBM achieves $4.4\%$ test error with no training, outperforming a $k=100$ randomly initialized RBM after 50 epochs of training.
• The HN-based initialization leads to faster convergence and better performance in both generative modeling and classification tasks, enabling smaller models to match or exceed the performance of larger, longer-trained models.