[Paper Review] AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methods

Zheng Shi, Abdurakhmon Sadiev+3 more|arXiv (Cornell University)|Feb 19, 2021

Stochastic Gradient Optimization Techniques34 references4 citations

TL;DR

This paper introduces AI-SARAH, a fully adaptive and tune-free stochastic recursive gradient method that implicitly adjusts step-sizes based on local geometry to improve convergence in convex finite-sum optimization. It estimates local Lipschitz smoothness dynamically, outperforms classical and state-of-the-art first-order methods without hyperparameter tuning, and provides theoretical convergence guarantees with intuitive design principles.

ABSTRACT

We present AI-SARAH, a practical variant of SARAH. As a variant of SARAH, this algorithm employs the stochastic recursive gradient yet adjusts step-size based on local geometry. AI-SARAH implicitly computes step-size and efficiently estimates local Lipschitz smoothness of stochastic functions. It is fully adaptive, tune-free, straightforward to implement, and computationally efficient. We provide technical insight and intuitive illustrations on its design and convergence. We conduct extensive empirical analysis and demonstrate its strong performance compared with its classical counterparts and other state-of-the-art first-order methods in solving convex machine learning problems.

Motivation & Objective

• To develop a practical, fully adaptive variant of SARAH that eliminates the need for manual hyperparameter tuning.
• To enable automatic step-size adaptation by implicitly estimating local Lipschitz smoothness of stochastic functions.
• To improve convergence performance in convex finite-sum optimization problems without requiring prior knowledge of problem parameters.
• To provide theoretical convergence analysis grounded in local geometry and recursive gradient estimation.
• To empirically validate superior performance against tuned classical and state-of-the-art first-order methods.

Proposed method

• AI-SARAH employs a stochastic recursive gradient estimator similar to SARAH, using the recursive update: $ v_t = \nabla f_i(w_t) - \nabla f_i(w_{t-1}) + v_{t-1} $.
• It dynamically adjusts the step-size $ \alpha_t $ based on local curvature, implicitly estimating the local Lipschitz constant $ L_k^{\max} $.
• The algorithm operates in outer and inner loops, with $ v_0 = \nabla P(w_{k-1}) $ at the start of each outer loop.
• It uses a working set $ \mathcal{W}_k $ defined by $ \|\tilde{w}_{k-1} - w\| \leq m \cdot \alpha_{\max}^k \|v_0\| $ to bound iterates and ensure stability.
• Theoretical analysis shows convergence via a recurrence bound on $ \mathbb{E}[\|\nabla P(\tilde{w}_k)\|^2] \leq \sigma_m^k \mathbb{E}[\|\nabla P(\tilde{w}_{k-1})\|^2] $, with $ \sigma_m^k $ depending on step-size and smoothness.
• The method is fully adaptive, requiring no manual tuning of learning rates or other hyperparameters.

Experimental results

Research questions

• RQ1 Can a stochastic recursive gradient method be made fully adaptive by implicitly estimating local smoothness without explicit hyperparameter tuning?
• RQ2 How does implicit step-size adaptation based on local geometry affect convergence speed and stability in convex finite-sum problems?
• RQ3 Can AI-SARAH achieve competitive performance against tuned first-order methods without any hyperparameter adjustment?
• RQ4 What theoretical guarantees can be established for adaptive step-sizes in recursive gradient methods under local smoothness assumptions?
• RQ5 How does the working set $ \mathcal{W}_k $ contribute to the stability and convergence of the algorithm?

Key findings

• AI-SARAH achieves competitive performance across multiple datasets without any hyperparameter tuning, outperforming SARAH, SARAH+, and other state-of-the-art methods that rely on fine-tuned parameters.
• The algorithm's convergence is guaranteed under standard assumptions of convexity and smoothness, with theoretical bounds derived via recurrence on gradient norm decay.
• Empirical results show consistent superiority over classical and adaptive methods, even when those methods are optimized over 5,000 hyperparameter runs.
• The implicit estimation of local Lipschitz smoothness enables effective step-size adaptation, improving convergence without explicit smoothness parameter input.
• The working set $ \mathcal{W}_k $ ensures that iterates remain within a bounded region, supporting theoretical stability and convergence.
• The method is computationally efficient, straightforward to implement, and fully adaptive—requiring zero tuning effort in practice.