[Paper Review] Towards a theory of machine learning

Vitaly Vanchurin|arXiv (Cornell University)|Apr 15, 2020

Statistical Mechanics and Entropy45 references4 citations

TL;DR

This paper proposes a statistical mechanics framework for machine learning by modeling neural networks as septuples with defined states, weights, biases, and loss functions. It derives thermodynamic laws of learning via maximum entropy and partition functions, showing that optimal learning efficiency arises in deep networks due to entropy and complexity trade-offs, with implications for emergent spacetime in fundamental physics.

ABSTRACT

We define a neural network as a septuple consisting of (1) a state vector, (2) an input projection, (3) an output projection, (4) a weight matrix, (5) a bias vector, (6) an activation map and (7) a loss function. We argue that the loss function can be imposed either on the boundary (i.e. input and/or output neurons) or in the bulk (i.e. hidden neurons) for both supervised and unsupervised systems. We apply the principle of maximum entropy to derive a canonical ensemble of the state vectors subject to a constraint imposed on the bulk loss function by a Lagrange multiplier (or an inverse temperature parameter). We show that in an equilibrium the canonical partition function must be a product of two factors: a function of the temperature and a function of the bias vector and weight matrix. Consequently, the total Shannon entropy consists of two terms which represent respectively a thermodynamic entropy and a complexity of the neural network. We derive the first and second laws of learning: during learning the total entropy must decrease until the system reaches an equilibrium (i.e. the second law), and the increment in the loss function must be proportional to the increment in the thermodynamic entropy plus the increment in the complexity (i.e. the first law). We calculate the entropy destruction to show that the efficiency of learning is given by the Laplacian of the total free energy which is to be maximized in an optimal neural architecture, and explain why the optimization condition is better satisfied in a deep network with a large number of hidden layers. The key properties of the model are verified numerically by training a supervised feedforward neural network using the method of stochastic gradient descent. We also discuss a possibility that the entire universe on its most fundamental level is a neural network.

Motivation & Objective

• To develop a unified theoretical framework for supervised and unsupervised learning using statistical mechanics principles.
• To address the lack of a fundamental explanation for the success of deep learning despite its high-dimensional parameter space.
• To define a loss function applicable in the bulk (hidden layers) rather than only on the boundary (input/output neurons), enabling unsupervised learning formalism.
• To derive equilibrium and non-equilibrium thermodynamic laws for neural network learning processes.
• To explore the possibility that the universe itself is fundamentally a neural network governed by similar principles.

Proposed method

• Defines a neural network as a septuple: state vector, input/output projections, weight matrix, bias vector, activation map, and loss function.
• Applies the principle of maximum entropy to derive a canonical ensemble of state vectors constrained by a bulk loss function via a Lagrange multiplier (inverse temperature).
• Calculates the canonical partition function as a product of a temperature-dependent factor and a function of weights and biases, enabling analytical treatment.
• Derives the first and second laws of learning: total entropy decreases to equilibrium, and loss increment is proportional to thermodynamic entropy and complexity increments.
• Introduces entropy destruction as a measure of learning efficiency, proportional to the Laplacian of the total free energy.
• Uses Gaussian integral approximation with smooth window functions to handle finite activation ranges, and defines an operator $\hat{G}$ whose spectrum determines the partition function.

Experimental results

Research questions

• RQ1 How can a unified thermodynamic description of machine learning be constructed for both supervised and unsupervised systems?
• RQ2 What is the role of the bulk (hidden-layer) loss function in enabling unsupervised learning within a statistical mechanics framework?
• RQ3 How does the canonical ensemble derived from maximum entropy relate to the equilibrium state of a neural network?
• RQ4 Why is learning efficiency higher in deep networks with many hidden layers, according to the derived thermodynamic model?
• RQ5 Can the emergence of spacetime and general relativity be derived from the non-equilibrium thermodynamics of neural networks?

Key findings

• The canonical partition function factorizes into a temperature-dependent term and a term depending on weights and biases, implying a decomposition of total free energy into thermodynamic and complexity components.
• The total Shannon entropy splits into thermodynamic entropy and a complexity term, both contributing to the learning process.
• The first law of learning states that the change in loss is proportional to the sum of changes in thermodynamic entropy and network complexity.
• The second law of learning holds as total entropy decreases during training until equilibrium is reached.
• Learning efficiency is maximized when the Laplacian of the total free energy is maximized, favoring deep architectures with many hidden layers.
• Under specific symmetry assumptions on the Onsager tensor, the entropy production leads to the Einstein field equations, suggesting that general relativity may emerge from neural network dynamics at large scales.