[Paper Review] Comparing Dynamics: Deep Neural Networks versus Glassy Systems

Marco Baity‐Jesi, Levent Sagun+7 more|Mar 19, 2018

Time Series Analysis and Forecasting20 citations

TL;DR

This paper compares the training dynamics of over-parametrized deep neural networks (DNNs) to glassy systems using statistical physics methods. It finds that DNNs exhibit slow, diffusive dynamics at the bottom of the loss landscape due to an increasing number of flat directions, with no barrier crossing—distinguishing them from mean-field glassy systems despite some similarities in aging behavior.

ABSTRACT

We analyze numerically the training dynamics of deep neural networks (DNN) by using methods developed in statistical physics of glassy systems. The two main issues we address are (1) the complexity of the loss landscape and of the dynamics within it, and (2) to what extent DNNs share similarities with glassy systems. Our findings, obtained for different architectures and datasets, suggest that during the training process the dynamics slows down because of an increasingly large number of flat directions. At large times, when the loss is approaching zero, the system diffuses at the bottom of the landscape. Despite some similarities with the dynamics of mean-field glassy systems, in particular, the absence of barrier crossing, we find distinctive dynamical behaviors in the two cases, showing that the statistical properties of the corresponding loss and energy landscapes are different. In contrast, when the network is under-parametrized we observe a typical glassy behavior, thus suggesting the existence of different phases depending on whether the network is under-parametrized or over-parametrized.

Motivation & Objective

• To investigate whether deep neural networks (DNNs) share dynamical properties with glassy systems, particularly in loss landscape exploration.
• To determine the role of over-parametrization in altering the statistical properties of the loss landscape compared to under-parametrized networks.
• To analyze the absence or presence of barrier crossing during DNN training using methods from statistical physics of disordered systems.
• To identify whether aging dynamics and slow relaxation in DNNs stem from glassy-like behavior or distinct mechanisms tied to over-parametrization.
• To explore the existence of phase transitions between easy and hard learning regimes based on network capacity and landscape structure.

Proposed method

• Numerical analysis of training dynamics in DNNs using stochastic gradient descent (SGD) across multiple architectures and datasets.
• Application of methods from statistical physics of glassy systems, including analysis of time correlation functions and mean-square displacement.
• Use of the time-dependent correlation function $\Delta(t_w, t_w + t)$ to detect aging behavior and distinguish between glassy and non-glassy dynamics.
• Comparison of loss landscape properties between over-parametrized and under-parametrized networks by reducing the number of nodes in model A.
• Investigation of the Edwards-Anderson parameter and trapping in local minima via collapse of mean-square displacement at small times.
• Examination of the shape of the correlation function $\Delta(t_w, t_w + t)$ at large $t_w$ to assess qualitative differences between DNNs and mean-field glassy systems.

Experimental results

Research questions

• RQ1 To what extent do the training dynamics of over-parametrized DNNs resemble those of mean-field glassy systems?
• RQ2 Does the absence of barrier crossing in DNNs indicate a fundamentally different statistical structure of the loss landscape compared to glassy systems?
• RQ3 How does over-parametrization affect the emergence of flat directions and the resulting slow dynamics in DNNs?
• RQ4 Is there a phase transition between an 'easy' learning phase (over-parametrized) and a 'hard' learning phase (under-parametrized) in DNNs?
• RQ5 Can the dynamical behavior of DNNs be explained by the presence of wide, flat basins of attraction rather than glassy trapping in local minima?

Key findings

• During training, DNNs slow down due to an increasing number of flat directions, leading to diffusive dynamics at the bottom of the loss landscape.
• Barrier crossing does not play a significant role in DNN training, consistent with the absence of deep local minima trapping the system.
• In over-parametrized networks, the system reaches near-zero loss and exhibits aging-like dynamics, but with distinct statistical properties compared to mean-field glassy systems.
• When the network is under-parametrized, the mean-square displacement shows a collapse at small times and $t_w$-dependent increase, indicating trapping and glassy aging behavior.
• The loss function in under-parametrized networks does not reach zero and tends asymptotically to a higher value, suggesting poor generalization or slow convergence.
• The authors conjecture a phase transition between an 'easy' phase (over-parametrized, flat landscape, fast learning) and a 'hard' phase (under-parametrized, rough landscape, glassy dynamics), analogous to phase transitions in combinatorial optimization.