[Paper Review] On the Convergence of SARAH and Beyond

Bingcong Li, Meng Ma+1 more|arXiv (Cornell University)|Jun 5, 2019

Stochastic Gradient Optimization Techniques39 references4 citations

TL;DR

This paper introduces L2S (Loopless SARAH), a novel variance reduction algorithm for minimizing finite-sum problems in machine learning. By leveraging a recursive gradient estimator and a unified analysis framework, L2S achieves optimal convergence rates: O((n+κ)ln(1/ε)) for strongly convex problems and O(n + √n/ε) for convex and nonconvex problems, with provable convergence using n-independent step sizes—resolving a key limitation of SARAH.

ABSTRACT

The main theme of this work is a unifying algorithm, extbf{L}oop extbf{L}ess extbf{S}ARAH (L2S) for problems formulated as summation of $n$ individual loss functions. L2S broadens a recently developed variance reduction method known as SARAH. To find an $ε$-accurate solution, L2S enjoys a complexity of ${\cal O}\big( (n+κ) \ln (1/ε)\big)$ for strongly convex problems. For convex problems, when adopting an $n$-dependent step size, the complexity of L2S is ${\cal O}(n+ \sqrt{n}/ε)$; while for more frequently adopted $n$-independent step size, the complexity is ${\cal O}(n+ n/ε)$. Distinct from SARAH, our theoretical findings support an $n$-independent step size in convex problems without extra assumptions. For nonconvex problems, the complexity of L2S is ${\cal O}(n+ \sqrt{n}/ε)$. Our numerical tests on neural networks suggest that L2S can have better generalization properties than SARAH. Along with L2S, our side results include the linear convergence of the last iteration for SARAH in strongly convex problems.

Motivation & Objective

• Address the lack of theoretical guarantees for SARAH with n-independent step sizes in convex problems.
• Provide a unified convergence analysis for variance reduction methods, particularly extending SARAH's applicability.
• Develop a new algorithm, L2S, that achieves optimal convergence rates without requiring loop structures or additional assumptions.
• Establish linear convergence of the last iterate for SARAH in strongly convex settings.
• Demonstrate improved generalization performance of L2S over SARAH in deep learning experiments.

Proposed method

• Propose L2S, a loopless variant of SARAH that uses a recursive gradient estimator without periodic full gradient computations.
• Introduce a novel analysis technique to bound the variance of the recursive gradient estimator using a telescoping sum argument.
• Derive convergence rates by analyzing the expected norm of the gradient and relating it to the initial error via a contraction argument.
• Use a step size policy that is independent of n for convex problems, removing the need for n-dependent tuning.
• Apply the method to both convex and nonconvex problems, including neural network training, with theoretical guarantees.
• Implement L2S with a uniform random selection of iterate from the inner loop to ensure convergence to an ε-accurate solution.

Experimental results

Research questions

• RQ1 Can a variance reduction method be designed to achieve optimal convergence without requiring loop structures or periodic full gradient evaluations?
• RQ2 Does L2S achieve the same convergence rate as SARAH while allowing n-independent step sizes in convex problems?
• RQ3 What is the convergence behavior of the last iterate of SARAH in strongly convex problems?
• RQ4 Can L2S outperform SARAH in terms of generalization in deep learning settings?
• RQ5 What is the optimal IFO complexity of L2S for convex, strongly convex, and nonconvex problems?

Key findings

• L2S achieves an IFO complexity of O((n + κ)ln(1/ε)) for strongly convex problems, matching the best-known rates.
• For convex problems, L2S attains O(n + √n/ε) complexity with an n-independent step size, which is a theoretical improvement over SARAH.
• L2S achieves O(n + √n/ε) complexity for nonconvex problems, matching the best-known rate for variance-reduced methods.
• The paper proves linear convergence of the last iterate of SARAH in the strongly convex case, a result not previously established.
• Numerical experiments on neural networks show that L2S generalizes better than SARAH, suggesting practical advantages.
• The analysis confirms that L2S does not require extra assumptions to support n-independent step sizes in convex problems, unlike prior work.