[Paper Review] SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradient

Lam M. Nguyen, Jie Liu+2 more|arXiv (Cornell University)|Feb 28, 2017

Stochastic Gradient Optimization Techniques267 citations

TL;DR

SARAH introduces a stochastic recursive gradient method for finite-sum minimization that uses past gradient information in a recursive update, achieves linear convergence for strongly convex problems, does not require storing past gradients, and includes a practical variant SARAH+ with adaptive inner-loop stopping.

ABSTRACT

In this paper, we propose a StochAstic Recursive grAdient algoritHm (SARAH), as well as its practical variant SARAH+, as a novel approach to the finite-sum minimization problems. Different from the vanilla SGD and other modern stochastic methods such as SVRG, S2GD, SAG and SAGA, SARAH admits a simple recursive framework for updating stochastic gradient estimates; when comparing to SAG/SAGA, SARAH does not require a storage of past gradients. The linear convergence rate of SARAH is proven under strong convexity assumption. We also prove a linear convergence rate (in the strongly convex case) for an inner loop of SARAH, the property that SVRG does not possess. Numerical experiments demonstrate the efficiency of our algorithm.

Motivation & Objective

• Motivate efficient optimization for large-scale finite-sum problems arising in supervised learning.
• Develop a variance-reducing stochastic gradient method that updates gradients recursively without storing all past gradients.
• Establish convergence guarantees: linear rates under strong convexity and sublinear rates in general convex settings.
• Provide a practical variant (SARAH+) with adaptive inner-loop stopping to improve robustness and performance.

Proposed method

• Introduce SARAH with outer loop full gradient evaluation and inner loop stochastic updates using v_t = ∇f_i(w_t) − ∇f_i(w_{t−1}) + v_{t−1} and w_{t+1} = w_t − η v_t.
• Show that SARAH is not an unbiased gradient estimator in general, but E[v_t] = E[∇P(w_t)], enabling convergence analysis.
• Prove linear convergence of the inner-loop gradient estimates under μ-strong convexity with appropriately chosen step size η.
• Provide convergence results for general convex and strongly convex cases, including sublinear rates for general convexity and linear rates with suitable parameter choices.
• Propose SARAH+ as a practical variant with adaptive inner-loop stopping based on ||v_t||^2 and an alternate finalization rule (t chosen as last inner index).
• Compare SARAH to SVRG, SAG/SAGA, and SGD variants in terms of storage, learning-rate requirements, and convergence guarantees.

Experimental results

Research questions

• RQ1 Can SARAH achieve linear convergence for strongly convex finite-sum problems while using low storage costs?
• RQ2 How does the recursive gradient update affect variance reduction and stability compared to SVRG and SAG/SAGA?
• RQ3 What are the convergence guarantees for general convex and strongly convex settings under SARAH, and how does SARAH+ improve practical performance?
• RQ4 What is the impact of inner-loop size m and learning rate η on convergence and robustness across datasets?

Key findings

• SARAH attains linear convergence in the strongly convex setting with a learning rate η = O(1/L) and inner-loop size m chosen relative to the condition number κ.
• The variance of inner-loop steps in SARAH decreases over time, yielding more stable updates than SVRG, and inner-loop linear convergence is established (Theorem 1a/1b).
• In the general convex case, SARAH achieves sublinear convergence and, with multiple outer iterations, achieves a total complexity of O((n + 1/ε) log(1/ε)).
• In the strongly convex case, SARAH achieves a total complexity of O((n + κ) log(1/ε)) similar to SVRG and SAG, with SARAH offering a smaller convergence-rate constant and greater stability.
• SARAH+ provides an adaptive inner-loop stopping criterion based on ||v_t||^2, enabling robust performance across datasets and often requiring less tuning than SVRG.
• Experimental results on multiple datasets show SARAH and SARAH+ outperform or match state-of-the-art first-order methods in terms of loss reduction and test accuracy.