[Paper Review] A Guide Through the Zoo of Biased SGD

Yury Demidovich, Grigory Malinovsky+2 more|arXiv (Cornell University)|May 25, 2023

Stochastic Gradient Optimization Techniques4 citations

TL;DR

This paper introduces a unified theoretical framework for analyzing Stochastic Gradient Descent (SGD) with biased gradient estimators, establishing connections among existing assumptions and proposing a new, weaker set of conditions that subsume all prior ones. It provides tight convergence guarantees in both convex and non-convex settings, demonstrating that biased estimators can outperform unbiased ones in practice, particularly in communication-constrained or black-box optimization scenarios.

ABSTRACT

Stochastic Gradient Descent (SGD) is arguably the most important single algorithm in modern machine learning. Although SGD with unbiased gradient estimators has been studied extensively over at least half a century, SGD variants relying on biased estimators are rare. Nevertheless, there has been an increased interest in this topic in recent years. However, existing literature on SGD with biased estimators (BiasedSGD) lacks coherence since each new paper relies on a different set of assumptions, without any clear understanding of how they are connected, which may lead to confusion. We address this gap by establishing connections among the existing assumptions, and presenting a comprehensive map of the underlying relationships. Additionally, we introduce a new set of assumptions that is provably weaker than all previous assumptions, and use it to present a thorough analysis of BiasedSGD in both convex and non-convex settings, offering advantages over previous results. We also provide examples where biased estimators outperform their unbiased counterparts or where unbiased versions are simply not available. Finally, we demonstrate the effectiveness of our framework through experimental results that validate our theoretical findings.

Motivation & Objective

• To address the lack of coherence in existing theoretical analyses of biased SGD, which rely on disconnected assumptions.
• To establish a comprehensive map of relationships among prior assumptions on biased gradient estimators.
• To propose a new, provably weaker set of assumptions that generalizes all previous ones.
• To provide tight convergence and complexity bounds for biased SGD in both convex and non-convex settings.
• To demonstrate empirical advantages of biased estimators over unbiased counterparts in real-world scenarios.

Proposed method

• Introduces a new assumption class that is strictly weaker than all prior assumptions on biased gradient estimators.
• Establishes formal relationships between existing assumption classes (e.g., bounded bias, contractive, absolute compression) using mathematical proofs.
• Derives convergence rates for biased SGD under the new assumption class using standard techniques like Lyapunov functions and Young’s inequality.
• Applies the framework to analyze specific biased estimators such as sparsified gradients, quantized updates, and zeroth-order methods.
• Uses error feedback mechanisms and compression operators to model practical communication-efficient training.
• Validates theoretical findings through experiments on deep learning and black-box optimization tasks.

Experimental results

Research questions

• RQ1 How are existing assumptions on biased gradient estimators related, and can they be unified under a single theoretical framework?
• RQ2 Can a new, weaker assumption be formulated that subsumes all prior assumptions and still guarantees convergence?
• RQ3 In what settings do biased estimators outperform unbiased ones, and why?
• RQ4 What are the tightest possible convergence rates for biased SGD in non-convex and convex settings under the new assumptions?
• RQ5 Can the proposed framework explain the success of practical methods like quantized SGD or zeroth-order optimization?

Key findings

• The proposed assumption class is strictly weaker than all prior assumptions, enabling broader applicability and tighter convergence analysis.
• The framework proves that biased estimators can achieve better convergence rates than their unbiased counterparts in certain settings, particularly with compression.
• Sparsified and quantized gradient estimators—common in distributed training—satisfy the new assumptions and are thus guaranteed to converge under the same conditions.
• Zeroth-order methods with biased gradient estimators are formally justified under the new framework, explaining their empirical success in black-box optimization.
• Empirical results confirm that biased SGD variants, such as those using error feedback, outperform unbiased baselines in terms of convergence speed and final accuracy on deep learning tasks.
• Theoretical analysis shows that the worst-case complexity of biased SGD under the new assumptions matches or improves upon existing bounds.