[Paper Review] Exponential Step Sizes for Non-Convex Optimization.

Xiaoyu Li, Zhenxun Zhuang+1 more|arXiv (Cornell University)|Feb 12, 2020

Stochastic Gradient Optimization Techniques23 references7 citations

TL;DR

This paper proposes exponential step sizes for stochastic optimization of smooth non-convex functions under the Polyak-Łojasiewicz (PL) condition, achieving adaptive convergence rates that interpolate between linear (noise-free) and $O(1/T)$ (noisy) behavior, up to poly-logarithmic factors. Without the PL condition, it still ensures optimal convergence to a critical point, up to logarithmic factors, validated on deep learning datasets.

ABSTRACT

Stochastic Gradient Descent (SGD) is a popular tool in large scale optimization of machine learning objective functions. However, the performance is greatly variable, depending on the choice of the step sizes. In this paper, we introduce the exponential step sizes for stochastic optimization of smooth non-convex functions which satisfy the Polyak-Łojasiewicz (PL) condition. We show that, without any information on the level of noise over the stochastic gradients, these step sizes guarantee a convergence rate for the last iterate that automatically interpolates between a linear rate (in the noisy-free case) and a $O(\frac{1}{T})$ rate (in the noisy case), up to poly-logarithmic factors. Moreover, if without the PL condition, the exponential step sizes still guarantee optimal convergence to a critical point, up to logarithmic factors. We also validate our theoretical results with empirical experiments on real-world datasets with deep learning architectures.

Motivation & Objective

• To address the high variability in stochastic gradient descent (SGD) performance due to step size selection in large-scale non-convex optimization.
• To develop step size strategies that adapt automatically to noise levels in stochastic gradients without requiring prior knowledge of noise.
• To establish convergence guarantees for the last iterate in non-convex settings, particularly under the Polyak-Łojasiewicz (PL) condition.
• To extend convergence results beyond the PL condition, ensuring optimal convergence to critical points up to logarithmic factors.

Proposed method

• Proposes exponential step sizes that decay geometrically over iterations, adapting to the optimization landscape without explicit noise estimation.
• Analyzes convergence under the Polyak-Łojasiewicz (PL) condition, showing that exponential steps achieve a convergence rate interpolating between linear and $O(1/T)$, up to poly-logarithmic factors.
• Derives theoretical bounds on the expected suboptimality gap for the last iterate, leveraging properties of smooth non-convex functions and stochastic gradients.
• Establishes that even without the PL condition, exponential step sizes lead to optimal convergence to a critical point, up to logarithmic factors.
• Employs a novel analysis framework that accounts for the interplay between step size decay and gradient noise variance.
• Validates the theoretical findings through empirical experiments on real-world deep learning datasets using standard architectures.

Experimental results

Research questions

• RQ1 Can exponential step sizes achieve adaptive convergence rates in non-convex stochastic optimization that respond to noise levels without prior knowledge?
• RQ2 How does the convergence rate of the last iterate behave under the Polyak-Łojasiewicz (PL) condition with exponential step sizes?
• RQ3 What convergence guarantees can be provided for exponential step sizes when the PL condition does not hold?
• RQ4 To what extent do exponential step sizes ensure optimal convergence to a critical point in non-convex settings?
• RQ5 How do exponential step sizes compare empirically to standard SGD in deep learning applications?

Key findings

• Exponential step sizes guarantee a convergence rate for the last iterate that interpolates between linear convergence (in the noise-free case) and $O(1/T)$ (in the noisy case), up to poly-logarithmic factors, under the PL condition.
• The method achieves optimal convergence to a critical point without requiring the PL condition, up to logarithmic factors.
• The convergence rate adapts automatically to the level of noise in stochastic gradients, without needing explicit noise estimation.
• Empirical results on real-world deep learning datasets confirm the theoretical advantages of exponential step sizes over standard SGD.
• The proposed step size strategy ensures robust performance across varying noise regimes, improving stability and convergence speed.