[Paper Review] High Probability Convergence of Clipped-SGD Under Heavy-tailed Noise

Ta Duy Nguyen, Thien Hang Nguyen+2 more|arXiv (Cornell University)|Feb 10, 2023

Stochastic Gradient Optimization Techniques4 citations

TL;DR

This paper establishes high-probability convergence bounds for clipped-SGD under heavy-tailed stochastic gradient noise with bounded $p$-th moments ($p \in (1,2]$), using a novel analysis framework that achieves time-optimal convergence rates up to logarithmic factors. The method requires only minimal assumptions and achieves noise-adaptive rates without momentum or bounded gradient assumptions.

ABSTRACT

While the convergence behaviors of stochastic gradient methods are well understood \emph{in expectation}, there still exist many gaps in the understanding of their convergence with \emph{high probability}, where the convergence rate has a logarithmic dependency on the desired success probability parameter. In the \emph{heavy-tailed noise} setting, where the stochastic gradient noise only has bounded $p$-th moments for some $p\in(1,2]$, existing works could only show bounds \emph{in expectation} for a variant of stochastic gradient descent (SGD) with clipped gradients, or high probability bounds in special cases (such as $p=2$) or with extra assumptions (such as the stochastic gradients having bounded non-central moments). In this work, using a novel analysis framework, we present new and time-optimal (up to logarithmic factors) \emph{high probability} convergence bounds for SGD with clipping under heavy-tailed noise for both convex and non-convex smooth objectives using only minimal assumptions.

Motivation & Objective

• Address the gap in theoretical understanding of stochastic first-order methods under heavy-tailed gradient noise, where traditional variance-based assumptions fail.
• Overcome the limitation of existing high-probability convergence results that rely on strong assumptions such as bounded non-central moments or domain diameter constraints.
• Establish time-optimal high-probability convergence for both convex and non-convex smooth objectives under minimal assumptions, specifically bounded $p$-th moment noise.
• Achieve noise-adaptive convergence rates that improve toward the deterministic rate as noise decreases, without requiring momentum or gradient normalization.
• Provide a general analysis framework applicable to both convex and non-convex settings, unifying convergence guarantees under heavy-tailed noise.

Proposed method

• Propose a clipped-SGD algorithm with a simple gradient clipping mechanism that does not use momentum.
• Introduce a novel concentration analysis framework to control the deviation of noisy iterates under heavy-tailed noise, relying only on bounded $p$-th moments.
• Use a high-probability martingale concentration inequality to bound the cumulative deviation of gradient norms, with a clipping threshold chosen to balance bias and variance.
• Derive a step size schedule that adapts to the noise level $\sigma$, the time horizon $T$, and the moment parameter $p$, ensuring optimal convergence rates.
• Employ a recursive decomposition of the iterates into bias and noise components, and apply a chaining argument to control the sum of squared deviations.
• Leverage a key lemma to bound the second moment of the deviation terms, enabling high-probability control via a carefully chosen $G$ parameter related to $\sigma^p$ and $T$.

Experimental results

Research questions

• RQ1 Can high-probability convergence of clipped-SGD be established under the minimal assumption of bounded $p$-th moment noise for $p \in (1,2]$?
• RQ2 Does the proposed method achieve time-optimal convergence rates (up to logarithmic factors) for both convex and non-convex objectives under heavy-tailed noise?
• RQ3 Can the convergence guarantee be noise-adaptive, improving as the noise level decreases, without requiring bounded gradients or non-central moments?
• RQ4 Is it possible to achieve high-probability convergence without relying on momentum or additional assumptions like bounded domain diameter?
• RQ5 Can a unified analysis framework be developed for both convex and non-convex settings under heavy-tailed noise?

Key findings

• The paper establishes high-probability convergence of clipped-SGD for both convex and non-convex smooth objectives under the minimal assumption of bounded $p$-th moment noise.
• The convergence rate is time-optimal up to logarithmic factors in $T$ and the success probability $\delta$, matching known lower bounds in expectation.
• The method achieves noise-adaptive rates: as the noise level $\sigma$ decreases, the convergence rate improves toward the deterministic rate.
• The analysis does not require bounded gradients, non-central moments, or domain diameter constraints, unlike prior works.
• The step size is chosen adaptively based on $\sigma$, $T$, $p$, and the problem's smoothness parameter $L$, ensuring optimal scaling.
• With high probability $1 - \delta$, the average squared gradient norm is bounded by $\mathcal{O}\left(\sqrt{\Delta_1 L} \ln \frac{4T}{\delta} \cdot T^{\frac{2-2p}{3p-2}} \max\left\{ \sigma^{\frac{p}{p-1}}, T^{\frac{1-2p}{3p-2}} \right\} \right)$, achieving the desired rate.