[Paper Review] Neural networks: from the perceptron to deep nets

Marylou Gabrié, Surya Ganguli+2 more|arXiv (Cornell University)|Apr 13, 2023

Neural Networks and Applications4 citations

TL;DR

This paper reviews the application of statistical mechanics—particularly the replica method and dynamical systems theory—to understand the generalization and training dynamics of neural networks from perceptrons to deep networks. It highlights how concepts like replica symmetry breaking, signal propagation, and dynamical isometry explain generalization in overparametrized models and enable training of extremely deep networks, while also linking non-equilibrium diffusion processes to modern generative AI models.

ABSTRACT

Artificial networks have been studied through the prism of statistical mechanics as disordered systems since the 80s, starting from the simple models of Hopfield's associative memory and the single-neuron perceptron classifier. Assuming data is generated by a teacher model, asymptotic generalisation predictions were originally derived using the replica method and the online learning dynamics has been described in the large system limit. In this chapter, we review the key original ideas of this literature along with their heritage in the ongoing quest to understand the efficiency of modern deep learning algorithms. One goal of current and future research is to characterize the bias of the learning algorithms toward well-generalising minima in a complex overparametrized loss landscapes with many solutions perfectly interpolating the training data. Works on perceptrons, two-layer committee machines and kernel-like learning machines shed light on these benefits of overparametrization. Another goal is to understand the advantage of depth while models now commonly feature tens or hundreds of layers. If replica computations apparently fall short in describing general deep neural networks learning, studies of simplified linear or untrained models, as well as the derivation of scaling laws provide the first elements of answers.

Motivation & Objective

• To trace the evolution of statistical mechanics approaches in understanding neural network generalization, from early models like the perceptron and Hopfield networks to modern deep learning.
• To explain how replica symmetry breaking and dynamical mean field theory elucidate the geometry of loss landscapes and the bias of optimization toward generalizing solutions.
• To investigate how signal propagation and initialization techniques (e.g., dynamical isometry) enable training of extremely deep networks, including 10,000-layer models.
• To connect non-equilibrium statistical mechanics to modern diffusion-based generative models, such as those used in text-to-image synthesis.
• To analyze the theoretical underpinnings of neural scaling laws and explore whether power-law scaling can be outperformed via intelligent data pruning.

Proposed method

• Applies the replica method to compute asymptotic generalization error in perceptrons and committee machines under high-dimensional limits.
• Uses dynamical mean field theory to analyze online and stochastic gradient descent dynamics in non-convex, high-dimensional loss landscapes.
• Employs free probability theory to compute the singular value spectrum of deep random weight matrices, enabling derivation of dynamically isometric initializations.
• Analyzes non-equilibrium diffusion processes in high-dimensional spaces to model generative modeling via score-based diffusion models.
• Applies replica calculations to non-Gaussian, pruned data distributions to study test error scaling beyond power laws.
• Derives scaling laws for model and data size using statistical mechanics models of learning, including generalization error as a function of system parameters.

Experimental results

Research questions

• RQ1 How does replica symmetry breaking explain the generalization capacity of non-convex classifiers in overparametrized settings?
• RQ2 What role does dynamical isometry play in enabling the training of extremely deep neural networks?
• RQ3 How can non-equilibrium statistical mechanics principles be used to design effective generative models for image synthesis?
• RQ4 What are the theoretical origins of observed neural scaling laws in terms of data, model size, and compute?
• RQ5 Can intelligent data pruning lead to scaling behaviors that exceed standard power-law scaling in test error?

Key findings

• The replica method predicts that a single-layer perceptron can store up to approximately 0.14N random patterns before generalization error increases sharply.
• Replica symmetry breaking reveals that the loss landscape of non-convex classifiers contains metastable states that correspond to good generalizing solutions.
• Dynamically isometric initializations, achieved via orthogonal weights and specific nonlinearities, allow training of 10,000-layer deep networks without normalization layers.
• Diffusion-based generative models are grounded in non-equilibrium statistical mechanics, where a neural network learns to reverse a noise-adding process to generate realistic images.
• Replica calculations show that with a good data pruning metric, test error can scale exponentially rather than as a power law, outperforming standard scaling laws.
• Neural scaling laws exhibit power-law decay of test error with respect to data size, model size, or compute, but these can be improved via structured data selection.