[Paper Review] Stochastic Recursive Gradient Algorithm for Nonconvex Optimization

Lam M. Nguyen, Jie Liu+2 more|arXiv (Cornell University)|May 20, 2017

Stochastic Gradient Optimization Techniques15 references67 citations

TL;DR

This paper analyzes mini-batch SARAH for nonconvex finite-sum problems, proving sublinear convergence for general nonconvex functions and linear convergence for gradient-dominated cases, with insights on mini-batch effects.

ABSTRACT

In this paper, we study and analyze the mini-batch version of StochAstic Recursive grAdient algoritHm (SARAH), a method employing the stochastic recursive gradient, for solving empirical loss minimization for the case of nonconvex losses. We provide a sublinear convergence rate (to stationary points) for general nonconvex functions and a linear convergence rate for gradient dominated functions, both of which have some advantages compared to other modern stochastic gradient algorithms for nonconvex losses.

Motivation & Objective

• Motivate efficient optimization for large-scale finite-sum nonconvex problems common in machine learning

Proposed method

• Proposes a mini-batch SARAH algorithm with outer and inner loops analogous to SVRG, but using a recursive gradient estimator
• Inner loop updates: v_t = (1/b) sum_{i in I_t} [∇f_i(w_t) − ∇f_i(w_{t-1})] + v_{t-1} with w_{t+1} = w_t − η v_t
• Full gradient is computed at the start of each outer loop; complexity per outer loop is O(n + bm) gradient evaluations
• Provides theoretical convergence analysis under L-smoothness (Assumption 1) and gradient dominance (Assumption 2)
• Derives sublinear convergence for SARAH-IN and linear convergence for gradient-dominated functions via η and m parameter choices
• Discusses the role of mini-batch size b on convergence, including corollaries showing b’s impact on rate and total complexity.

Experimental results

Research questions

• RQ1 What convergence rates does mini-batch SARAH achieve for general nonconvex objectives?
• RQ2 Under what conditions does SARAH enjoy linear convergence for gradient-dominated nonconvex functions?
• RQ3 How does mini-batch size affect convergence and complexity bounds for SARAH?
• RQ4 How does SARAH compare with SGD, SVRG, and GD in theory and practice for nonconvex empirical loss minimization?
• RQ5 What practical considerations arise for implementing SARAH and its variants (e.g., SARAH+) on neural networks?

Key findings

• SARAH-IN achieves sublinear convergence in expectation for general nonconvex P with appropriate η and inner loop length m
• For gradient-dominated (τ-gradient dominated) P, SARAH attains linear convergence to a global minimum under suitable η and m, with rates depending on τ and L
• The total IFO complexity to reach ε-accuracy is O(n + L^2/ε^2) in the general nonconvex setting, and O((n + L^2 τ^2) log(1/ε)) for gradient-dominated cases
• Mini-batch size b influences the allowable learning rate and inner loop size, with larger b enabling faster practical convergence
• The practical SARAH+ variant uses adaptive inner-loop termination and performs competitively against SVRG and SGD-based methods on neural networks (MNIST, CIFAR-10)