[Paper Review] Mean-field theory of learning: from dynamics to statics

K. Y. Michael Wong, S. Li+1 more|arXiv (Cornell University)|Jun 15, 2000

Neural Networks and Applications3 references3 citations

TL;DR

This paper develops a mean-field theory for batch learning in neural networks by combining the cavity method and diagrammatic techniques to model learning dynamics from first principles. It derives self-consistent equations for macroscopic observables like overlap and activation distributions, showing convergence to steady states consistent with static cavity theory, and provides a framework for analyzing rough energy landscapes in large networks with temporal correlations.

ABSTRACT

Using the cavity method and diagrammatic methods, we model the dynamics of batch learning of restricted sets of examples. Simulations of the Green's function and the cavity activation distributions support the theory well. The learning dynamics approaches a steady state in agreement with the static version of the cavity method. The picture of the rough energy landscape is reviewed.

Motivation & Objective

• To develop a theoretical framework for batch learning dynamics in large neural networks with temporal correlations among examples.
• To bridge dynamic learning processes with static equilibrium properties using mean-field approximations.
• To analyze the role of rough energy landscapes in learning, particularly when local minima emerge due to instability.
• To generalize the cavity method beyond equilibrium to describe time-evolving learning processes.
• To establish a connection between dynamic observables (e.g., Green’s functions) and static quantities (e.g., TAP equations) via diagrammatic techniques.

Proposed method

• Uses the cavity method to compare activation distributions when a training example is present versus absent in the training set.
• Applies linear response theory and constructs Green’s functions to describe the propagation of influence from added examples through learning history.
• Employs diagrammatic expansions with pairing rules to compute averages over examples, extending methods used in linear learning to nonlinear rules.
• Derives self-consistent equations for key observables: $q_0$, $q_1$, $\chi$, $\gamma$, and $w$, using cavity and free energy arguments.
• Introduces a non-equilibrium free energy $F(p,N)$ to characterize the density of local minima and their energy distribution.
• Solves the system of equations under the assumption of extensive free energy scaling, linking network size and example count.

Experimental results

Research questions

• RQ1 How do temporal correlations in batch learning affect the convergence to steady states in large neural networks?
• RQ2 Can the cavity method be extended from static equilibrium to dynamic learning processes?
• RQ3 What is the role of the energy landscape roughness in determining learning dynamics and stability?
• RQ4 How do macroscopic observables like weight overlaps and activation variances evolve during batch learning?
• RQ5 What is the relationship between dynamic Green’s functions and static TAP equations in the steady state?

Key findings

• The learning dynamics converge to a steady state that matches the predictions of the static cavity method, validating the theoretical framework.
• The cavity activation distribution and Green’s function simulations support the theoretical predictions, confirming the accuracy of the mean-field approximation.
• The theory correctly captures the influence of temporal correlations in batch learning, which are absent in on-line learning models.
• A set of five self-consistent equations is derived for $\chi$, $\gamma$, $q_1$, $q_0$, and $w$, describing the full dynamic and static behavior.
• The stability of the solution is governed by a condition equivalent to the Almeida-Thouless condition, beyond which local minima appear and the landscape becomes rough.
• The free energy scaling argument ensures that $F(p,N)$ is extensive, enabling the derivation of the density of states and energy distribution of local minima.