[Paper Review] On Variance Reduction in Stochastic Gradient Descent and its Asynchronous Variants

Sashank J. Reddi, Ahmed Hefny+3 more|arXiv (Cornell University)|Jun 23, 2015

Stochastic Gradient Optimization Techniques27 references111 citations

TL;DR

This paper proposes a unified framework for variance-reduced stochastic gradient descent (SGD) algorithms, enabling the development of asynchronous variants that achieve near-linear speedup in sparse machine learning settings. It introduces an asynchronous SVRG-like algorithm with provably linear convergence, outperforming both standard SGD and non-variance-reduced asynchronous methods in practice.

ABSTRACT

We study optimization algorithms based on variance reduction for stochastic gradient descent (SGD). Remarkable recent progress has been made in this direction through development of algorithms like SAG, SVRG, SAGA. These algorithms have been shown to outperform SGD, both theoretically and empirically. However, asynchronous versions of these algorithms---a crucial requirement for modern large-scale applications---have not been studied. We bridge this gap by presenting a unifying framework for many variance reduction techniques. Subsequently, we propose an asynchronous algorithm grounded in our framework, and prove its fast convergence. An important consequence of our general approach is that it yields asynchronous versions of variance reduction algorithms such as SVRG and SAGA as a byproduct. Our method achieves near linear speedup in sparse settings common to machine learning. We demonstrate the empirical performance of our method through a concrete realization of asynchronous SVRG.

Motivation & Objective

• To address the lack of theoretical convergence guarantees for asynchronous variance-reduced SGD in large-scale machine learning.
• To unify existing variance reduction techniques like SAG, SVRG, and SAGA under a single formal framework.
• To develop an asynchronous parallel algorithm grounded in this framework that maintains fast convergence.
• To demonstrate empirical speedups in sparse data regimes, validating theoretical near-linear speedup.
• To extend the applicability of variance reduction to modern distributed and asynchronous computing environments.

Proposed method

• The authors propose a general algorithmic framework that maintains auxiliary variables α_i^t for each component function f_i, enabling variance reduction through structured gradient updates.
• The framework incorporates a ScheduleUpdate subroutine that determines how auxiliary variables are updated, allowing instantiation of known methods like SVRG and SAGA.
• An asynchronous variant, Hsag, is derived from the framework, using delayed gradients and lock-free updates to enable parallel execution.
• The method employs a step size η, momentum parameter β, and regularization parameter c, with parameters tuned to ensure linear convergence.
• Convergence analysis relies on bounding the expected squared norm of the gradient estimate using Bregman divergences and variance decomposition.
• Theoretical guarantees are derived under strong convexity and Lipschitz gradient assumptions, showing linear convergence in expectation.

Experimental results

Research questions

• RQ1 Can a unified framework be developed to systematically describe and analyze variance-reduced SGD methods like SAG, SVRG, and SAGA?
• RQ2 Can asynchronous variants of these variance-reduced methods be designed with provable linear convergence?
• RQ3 Do these asynchronous algorithms achieve near-linear speedup in sparse machine learning settings?
• RQ4 How does the performance of the proposed asynchronous variance-reduced method compare to non-variance-reduced asynchronous SGD?
• RQ5 What parameter settings ensure fast convergence in both synchronous and asynchronous settings?

Key findings

• The proposed Hsag algorithm achieves linear convergence for smooth, strongly convex finite-sum problems under asynchronous updates.
• In the high condition number regime (L/λ = n), the algorithm attains ε-accuracy in O(n log(1/ε)) total computational complexity, matching the theoretical lower bound of batch gradient descent but with better practical performance.
• For sparse data settings, the method achieves near-linear speedup with respect to the number of processors, as predicted by theory.
• Empirical results show strong speedups over non-variance-reduced asynchronous SGD, confirming the theoretical advantage of variance reduction in parallel settings.
• The framework naturally yields asynchronous versions of SVRG and SAGA as special cases, demonstrating its generality and extensibility.
• Sharper convergence constants can be derived through refined analysis, as shown in the case of SVRG and SAGA, indicating room for further optimization.