[Paper Review] SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality

Courtney Paquette, Kiwon Lee+2 more|arXiv (Cornell University)|Feb 8, 2021

Stochastic Gradient Optimization Techniques76 references4 citations

TL;DR

This paper introduces a novel average-case analysis framework for stochastic gradient descent (SGD) in high-dimensional settings using random matrix theory. It shows that with fixed stepsizes, SGD dynamics on random least-squares problems become deterministic and governed by a Volterra integral equation, revealing a critical stepsize at which convergence behavior undergoes a phase transition, with explicit average-case convergence rates significantly outperforming worst-case bounds.

ABSTRACT

We propose a new framework, inspired by random matrix theory, for analyzing the dynamics of stochastic gradient descent (SGD) when both number of samples and dimensions are large. This framework applies to any fixed stepsize and the finite sum setting. Using this new framework, we show that the dynamics of SGD on a least squares problem with random data become deterministic in the large sample and dimensional limit. Furthermore, the limiting dynamics are governed by a Volterra integral equation. This model predicts that SGD undergoes a phase transition at an explicitly given critical stepsize that ultimately affects its convergence rate, which we also verify experimentally. Finally, when input data is isotropic, we provide explicit expressions for the dynamics and average-case convergence rates (i.e., the complexity of an algorithm averaged over all possible inputs). These rates show significant improvement over the worst-case complexities.

Motivation & Objective

• To develop a new theoretical framework for analyzing SGD dynamics when both data size and model dimension are large.
• To understand the impact of fixed stepsizes on SGD convergence, especially in the non-asymptotic, finite-sum setting.
• To derive average-case convergence rates that reflect empirical performance more accurately than worst-case bounds.
• To identify a critical stepsize at which SGD dynamics undergo a phase transition, affecting convergence speed.

Proposed method

• Uses random matrix theory to model the Hessian's eigenvalue distribution in high-dimensional least-squares problems.
• Derives a deterministic Volterra integral equation governing the limiting dynamics of SGD with fixed stepsize.
• Characterizes the solution of the Volterra equation as a function of the stepsize, forcing function, and eigenvalue measure.
• Establishes weak convergence of empirical spectral measures to a limiting distribution under high-dimensional limits.
• Applies the method to isotropic and one-hidden-layer data models to derive explicit average-case convergence rates.
• Validates predictions via numerical simulations comparing the Volterra model to SDE, SME, and streaming models.

Experimental results

Research questions

• RQ1 How do SGD dynamics behave in the large-sample, high-dimensional limit with a fixed stepsize?
• RQ2 Can the dynamics of SGD be described by a deterministic equation in the finite-sum, non-asymptotic regime?
• RQ3 Does a critical stepsize exist where the convergence behavior of SGD undergoes a phase transition?
• RQ4 What are the average-case convergence rates for SGD under isotropic data, and how do they compare to worst-case rates?
• RQ5 How well does the proposed Volterra model predict actual SGD performance compared to existing stochastic models?

Key findings

• The dynamics of SGD on random least-squares problems converge to the solution of a Volterra integral equation in the large-sample, high-dimensional limit.
• A phase transition occurs at a critical stepsize, which is explicitly derived and shown to affect convergence rate discontinuously.
• For isotropic data, the average-case convergence rate is significantly better than the worst-case rate, with explicit closed-form expressions provided.
• The Volterra model accurately tracks SGD performance across various stepsizes, outperforming SDE, SME, and streaming models in fidelity.
• The critical stepsize threshold is derived as a function of the ratio of parameters to samples and the Hessian’s eigenvalue distribution.
• Numerical experiments confirm the theoretical predictions, showing the Volterra model matches SGD dynamics while other models introduce significant bias.