[Paper Review] The Dynamics of Learning: A Random Matrix Approach

Zhenyu Liao, Romain Couillet|arXiv (Cornell University)|May 30, 2018

Neural Networks and Applications17 citations

TL;DR

This paper introduces a random matrix theory-based framework to analyze the learning dynamics of single-layer linear networks trained via gradient descent on high-dimensional binary classification data. It reveals that early stopping prevents overfitting when data dimension and sample size are comparable, and shows random initialization can degrade performance, offering theoretical insights into generalization and optimization in deep networks.

ABSTRACT

Understanding the learning dynamics of neural networks is one of the key issues for the improvement of optimization algorithms as well as for the theoretical comprehension of why deep neural nets work so well today. In this paper, we introduce a random matrix-based framework to analyze the learning dynamics of a single-layer linear network on a binary classification problem, for data of simultaneously large dimension and size, trained by gradient descent. Our results provide rich insights into common questions in neural nets, such as overfitting, early stopping and the initialization of training, thereby opening the door for future studies of more elaborate structures and models appearing in today's neural networks.

Motivation & Objective

• To understand the generalization and optimization behavior of neural networks in high-dimensional, overparameterized settings where parameters exceed training samples.
• To analyze how gradient descent training dynamics affect generalization, particularly in the presence of overfitting.
• To investigate the role of initialization and early stopping in controlling model performance.
• To develop a theoretical framework using random matrix theory applicable to real-world deep learning scenarios.

Proposed method

• Models a single-layer linear network trained via full-batch gradient descent on binary classification data with Gaussian-distributed classes centered at ±μ and identity covariance.
• Uses random matrix theory to derive deterministic equivalents for key performance metrics like training and generalization error.
• Analyzes the evolution of the weight vector w(t) over time t, showing convergence to a direction aligned with μ despite random initialization.
• Derives closed-form expressions for the expected signal-to-noise ratio E/√V, which quantifies generalization performance.
• Applies complex contour integrals and real integral representations to compute eigenvalue functionals of the sample covariance matrix.
• Validates theoretical predictions through simulations on MNIST data with n = p = 784, showing strong agreement between theory and empirical results.

Experimental results

Research questions

• RQ1 How does the generalization performance of a linear network evolve during gradient descent training in high-dimensional settings?
• RQ2 What is the impact of random weight initialization on the final model performance in overparameterized regimes?
• RQ3 Under what conditions does overfitting emerge, and can early stopping effectively mitigate it?
• RQ4 What is the theoretical lower bound on training sample size required for reliable classification in this setting?
• RQ5 How do the eigenvalues of the data covariance matrix influence learning dynamics and generalization?

Key findings

• Early stopping effectively prevents overfitting when the number of training samples is close to the data dimension, especially under high-dimensional settings.
• When the data dimension p equals the number of samples n, overfitting becomes severe, and generalization performance degrades significantly if training continues past the optimal stopping time.
• Random initialization leads to a suboptimal signal-to-noise ratio E/√V < ||μ||, and although the network eventually converges to the optimal direction, performance degrades initially.
• The theoretical framework accurately predicts training and generalization performance, with strong empirical validation on MNIST data (n = p = 784) across 100 runs.
• For c = 0 (no explicit regularization), generalization performance continues to improve with training time, indicating no overtraining in this regime.
• A strict lower bound on the number of training samples is derived for a given classification task, ensuring reliable learning in the high-dimensional limit.