[Paper Review] High probability convergence and uniform stability bounds for nonconvex stochastic gradient descent

Liam Madden, Emiliano Dall’Anese+1 more|arXiv (Cornell University)|Jun 10, 2020

Stochastic Gradient Optimization Techniques4 citations

TL;DR

This paper establishes high probability convergence and uniform stability bounds for nonconvex stochastic gradient descent (SGD) under smoothness, Polyak-Łojasiewicz inequality, and bounded variation assumptions. It derives epoch-dependent bounds that balance convergence and generalization, showing the true risk vanishes as dataset size increases, thus bridging theory and practice in nonconvex optimization.

ABSTRACT

Stochastic gradient descent (with a mini-batch) is one of the most common iterative algorithms used in machine learning. While being computationally cheap to implement, recent literature suggests that it may also have implicit regularization properties that prevent overfitting. This paper analyzes the properties of stochastic gradient descent from a theoretical standpoint to help bridge the gap between theoretical and empirical results; in particular, we prove bounds that depend explicitly on the number of epochs. Assuming smoothness, the Polyak-Łojasiewicz inequality, and the bounded variation property, we prove high probability bounds on the convergence rate. Assuming Lipschitz continuity and smoothness, we prove high probability bounds on the uniform stability. Putting these together (noting that some of the assumptions imply each other), we bound the true risk of the iterates of stochastic gradient descent. For convergence, our high probability bounds match existing expected bounds. For stability, our high probability bounds extend the nonconvex expected bound in Hardt et al. (2015). We use existing results to bound the generalization error using the stability. Finally, we put the convergence and generalization bounds together. We find that for a certain number of epochs of stochastic gradient descent, the convergence and generalization balance and we get a true risk bound that goes to zero as the number of samples goes to infinity.

Motivation & Objective

• To close the gap between theoretical analysis and empirical success of stochastic gradient descent (SGD) in nonconvex machine learning problems.
• To derive high probability convergence bounds for SGD that explicitly depend on the number of training epochs.
• To extend existing expected stability bounds to high probability uniform stability in nonconvex settings.
• To combine convergence and generalization bounds to establish a vanishing true risk under increasing sample size.

Proposed method

• Leverages smoothness, Polyak-Łojasiewicz (PL) inequality, and bounded variation of gradients to derive high probability convergence rates for SGD iterates.
• Applies Lipschitz continuity and smoothness assumptions to establish high probability uniform stability bounds for SGD.
• Uses existing generalization error bounds based on uniform stability to link stability to true risk.
• Combines convergence and stability bounds via a trade-off in the number of epochs to achieve diminishing true risk.
• Employs concentration inequalities and martingale arguments to derive high probability bounds, avoiding reliance on expectation-only analysis.
• Derives epoch-dependent bounds that reflect practical training schedules, making theoretical results more actionable.

Experimental results

Research questions

• RQ1 Can high probability convergence bounds for nonconvex SGD be derived under standard assumptions like smoothness and PL inequality?
• RQ2 How do high probability uniform stability bounds for nonconvex SGD compare to existing expected stability bounds?
• RQ3 What is the interplay between convergence and generalization in SGD when the number of epochs is explicitly considered?
• RQ4 Can the combination of convergence and stability bounds yield a true risk bound that vanishes as the number of samples increases?
• RQ5 Do the assumptions used in the analysis imply each other, and how does this affect the tightness of the derived bounds?

Key findings

• The paper establishes high probability convergence bounds for nonconvex SGD that match existing expected bounds in rate, but with probabilistic guarantees.
• It extends the nonconvex expected stability bound of Hardt et al. (2015) to a high probability uniform stability bound under similar assumptions.
• The true risk of SGD iterates is bounded and decays to zero as the number of training samples increases, under the derived convergence and stability conditions.
• The balance between convergence and generalization is achieved for a specific number of epochs, leading to optimal generalization performance.
• The derived bounds depend explicitly on the number of epochs, making them more interpretable and practical for real-world training schedules.
• Assumptions such as smoothness and bounded variation are shown to be sufficient to derive both convergence and stability results, with some implying each other in the analysis.