[Paper Review] Edge of chaos as a guiding principle for modern neural network training

Lin Zhang, Ling Feng+2 more|arXiv (Cornell University)|Jul 20, 2021

Neural Networks and Applications36 references4 citations

TL;DR

This paper proposes that training deep neural networks at the edge of chaos—where dynamical systems exhibit optimal computational capabilities—can guide hyperparameter selection for faster, more robust training. Using stochastic gradient descent with momentum on a simple feedforward network, the authors identify scaling relations in the ordered phase that allow accelerated training without performance loss, and show that weight decay effectively pushes models toward the edge of chaos, enabling optimal generalization.

ABSTRACT

The success of deep neural networks in real-world problems has prompted many attempts to explain their training dynamics and generalization performance, but more guiding principles for the training of neural networks are still needed. Motivated by the edge of chaos principle behind the optimal performance of neural networks, we study the role of various hyperparameters in modern neural network training algorithms in terms of the order-chaos phase diagram. In particular, we study a fully analytical feedforward neural network trained on the widely adopted Fashion-MNIST dataset, and study the dynamics associated with the hyperparameters in back-propagation during the training process. We find that for the basic algorithm of stochastic gradient descent with momentum, in the range around the commonly used hyperparameter values, clear scaling relations are present with respect to the training time during the ordered phase in the phase diagram, and the model's optimal generalization power at the edge of chaos is similar across different training parameter combinations. In the chaotic phase, the same scaling no longer exists. The scaling allows us to choose the training parameters to achieve faster training without sacrificing performance. In addition, we find that the commonly used model regularization method - weight decay - effectively pushes the model towards the ordered phase to achieve better performance. Leveraging on this fact and the scaling relations in the other hyperparameters, we derived a principled guideline for hyperparameter determination, such that the model can achieve optimal performance by saturating it at the edge of chaos. Demonstrated on this simple neural network model and training algorithm, our work improves the understanding of neural network training dynamics, and can potentially be extended to guiding principles of more complex model architectures and algorithms.

Motivation & Objective

• To investigate whether the edge of chaos principle can guide hyperparameter selection in modern neural network training.
• To understand how training algorithms like stochastic gradient descent with momentum influence the system's dynamical phase (ordered vs. chaotic).
• To identify scaling relations in hyperparameters during training that enable faster convergence without sacrificing performance.
• To explore how regularization techniques such as weight decay affect the system's proximity to the edge of chaos.
• To derive a principled guideline for hyperparameter tuning that saturates the model at the edge of chaos for optimal generalization.

Proposed method

• The study uses a fully analytical, single-hidden-layer feedforward neural network trained on the Fashion-MNIST dataset.
• The order-chaos phase diagram is constructed by analyzing the asymptotic stability of the network's dynamics using dynamical mean-field theory (DMFT).
• The edge of chaos is identified via the condition $ \frac{J^2}{N} \sum_i \text{sech}^4(\cdots) = 1 $, derived from the variance of weight updates.
• Scaling relations between hyperparameters (learning rate, momentum, weight decay) and training duration (epochs) are identified in the ordered phase.
• Weight decay is modeled as a negative feedback on weight variance, with its effect quantified via iterative substitution of the SGD update rule.
• Theoretical predictions are validated numerically by perturbing test inputs and measuring asymptotic distance distributions across epochs.

Experimental results

Research questions

• RQ1 Do scaling relations between hyperparameters and training duration exist in the ordered phase of stochastic gradient descent with momentum?
• RQ2 How does weight decay influence the system's position in the order-chaos phase diagram?
• RQ3 Can the edge of chaos be systematically reached through regularization, and does this lead to optimal generalization?
• RQ4 Is the asymptotic behavior of the network a global property independent of input data?
• RQ5 Can the observed scaling relations be leveraged to accelerate training without performance degradation?

Key findings

• Clear scaling relations exist between hyperparameters and training duration in the ordered phase, enabling faster training without performance loss.
• The model's generalization performance remains nearly invariant across different hyperparameter combinations within the scaling regime, indicating robustness.
• In the chaotic phase, such scaling relations break down, confirming the ordered phase as the optimal training regime.
• Weight decay effectively reduces the effective weight variance, pushing the model toward the ordered phase and enabling control over its position near the edge of chaos.
• Theoretical analysis shows that weight decay reduces $ J^2 $ by a factor of $ 4\eta\lambda / (1 - \alpha) $, allowing precise tuning to reach the edge of chaos.
• By equating the incremental increase in $ J^2 $ from optimization steps with the decrement from weight decay, the authors derive a principled formula for hyperparameter selection that saturates the model at the edge of chaos.