[Paper Review] An exact mapping between the Variational Renormalization Group and Deep Learning

Pankaj Mehta, David J. Schwab|arXiv (Cornell University)|Oct 14, 2014

Quantum many-body systems2 references142 citations

TL;DR

This paper establishes an exact mathematical mapping between the variational renormalization group (VRG) in statistical physics and deep learning architectures based on Restricted Boltzmann Machines (RBMs). It demonstrates that deep neural networks naturally implement a generalized coarse-graining procedure analogous to Kadanoff block spin renormalization, revealing that deep learning's success in feature extraction may stem from its intrinsic renormalization-like structure, validated analytically in 1D and numerically in 2D Ising models.

ABSTRACT

Deep learning is a broad set of techniques that uses multiple layers of representation to automatically learn relevant features directly from structured data. Recently, such techniques have yielded record-breaking results on a diverse set of difficult machine learning tasks in computer vision, speech recognition, and natural language processing. Despite the enormous success of deep learning, relatively little is understood theoretically about why these techniques are so successful at feature learning and compression. Here, we show that deep learning is intimately related to one of the most important and successful techniques in theoretical physics, the renormalization group (RG). RG is an iterative coarse-graining scheme that allows for the extraction of relevant features (i.e. operators) as a physical system is examined at different length scales. We construct an exact mapping from the variational renormalization group, first introduced by Kadanoff, and deep learning architectures based on Restricted Boltzmann Machines (RBMs). We illustrate these ideas using the nearest-neighbor Ising Model in one and two-dimensions. Our results suggests that deep learning algorithms may be employing a generalized RG-like scheme to learn relevant features from data.

Motivation & Objective

• To understand the theoretical basis for the success of deep learning in unsupervised feature learning and data compression.
• To investigate whether deep neural networks (DNNs) perform a form of iterative coarse-graining similar to the renormalization group (RG) in physics.
• To establish a precise, exact mapping between the variational renormalization group (VRG) and deep learning architectures based on Restricted Boltzmann Machines (RBMs).
• To demonstrate that DNNs trained on spin systems self-organize into a structure resembling Kadanoff block spin renormalization.
• To explore whether concepts from advanced RG techniques—such as fixed points, universality, and entanglement—can inform or improve deep learning models.

Proposed method

• Construct an exact mapping between the variational renormalization group (VRG) framework, originally developed by Kadanoff, and deep learning models based on Restricted Boltzmann Machines (RBMs).
• Use a variational procedure in VRG to minimize the free energy difference between physical and coarse-grained systems, analogous to minimizing the Kullback-Leibler divergence in RBMs.
• Apply the mapping to the 1D and 2D nearest-neighbor Ising models, using analytical methods for the 1D case and numerical training of stacked RBMs for the 2D case.
• Define effective receptive fields using recursive convolution of weight matrices, tracking how visible-layer spins influence hidden-layer neurons.
• Train stacked RBMs using contrastive divergence with L1 regularization and momentum, optimizing for unsupervised feature learning on Ising model data.
• Visualize the effective receptive fields to confirm that hidden units in deeper layers integrate information from increasingly larger regions of the visible layer, mimicking RG coarse-graining.

Experimental results

Research questions

• RQ1 Is there an exact mathematical correspondence between the variational renormalization group and deep learning architectures such as RBMs?
• RQ2 Do deep neural networks used in unsupervised learning perform a form of iterative coarse-graining similar to the renormalization group in statistical physics?
• RQ3 Can the feature learning process in deep networks be interpreted as a generalized renormalization group transformation that preserves long-distance physics?
• RQ4 How do the learned representations in deep networks compare to those generated by Kadanoff block spin transformations in the Ising model?
• RQ5 To what extent can advanced RG concepts—such as fixed points, universality, and entanglement entropy—be transferred to improve or understand deep learning models?

Key findings

• An exact one-to-one mapping exists between the variational renormalization group and deep learning architectures based on Restricted Boltzmann Machines (RBMs).
• The deep neural network trained on the 1D Ising model exactly reproduces the Kadanoff block spin transformation, confirming the theoretical mapping.
• In the 2D Ising model, stacked RBMs self-organize to implement a coarse-graining process that closely resembles Kadanoff block spin renormalization, as evidenced by the structure of effective receptive fields.
• The effective receptive fields of hidden units grow progressively larger with depth, indicating that deeper layers integrate information from increasingly larger spatial regions of the visible layer.
• The learned weight matrices in the DNN exhibit a hierarchical structure where each layer captures increasingly abstract, long-range correlations, mirroring the RG flow of relevant operators.
• The use of L1 regularization in training prevents all-to-all coupling and enforces sparsity, which helps maintain a meaningful coarse-graining hierarchy, consistent with RG principles.