[論文レビュー] Stochastic Recursive Gradient Algorithm for Nonconvex Optimization

Lam M. Nguyen, Jie Liu+2 more|arXiv (Cornell University)|May 20, 2017

Stochastic Gradient Optimization Techniques参考文献 15被引用数 67

ひとこと要約

本論文は非凸な有限和問題に対するミニバッチ SARAH を分析し、一般の非凸関数に対してサブ線形収束を、勾配支配の場合には線形収束を証明し、ミニバッチの効果に関する洞察を示す。

ABSTRACT

In this paper, we study and analyze the mini-batch version of StochAstic Recursive grAdient algoritHm (SARAH), a method employing the stochastic recursive gradient, for solving empirical loss minimization for the case of nonconvex losses. We provide a sublinear convergence rate (to stationary points) for general nonconvex functions and a linear convergence rate for gradient dominated functions, both of which have some advantages compared to other modern stochastic gradient algorithms for nonconvex losses.

研究の動機と目的

• 機械学習で一般的な大規模な有限和非凸問題の効率的最適化を動機づける

提案手法

• SVRG に類比する外部ループと内部ループを備えたミニバッチ SARAH アルゴリズムを提案するが、勾配推定量を再帰的に用いる。
• Inner loop updates: v_t = (1/b) sum_{i in I_t} [∇f_i(w_t) − ∇f_i(w_{t-1})] + v_{t-1} with w_{t+1} = w_t − η v_t
• Full gradient is computed at the start of each outer loop; complexity per outer loop is O(n + bm) gradient evaluations
• Provides theoretical convergence analysis under L-smoothness (Assumption 1) and gradient dominance (Assumption 2)
• Derives sublinear convergence for SARAH-IN and linear convergence for gradient-dominated functions via η and m parameter choices
• Discusses the role of mini-batch size b on convergence, including corollaries showing b’s impact on rate and total complexity.

実験結果

リサーチクエスチョン

• RQ1 What convergence rates does mini-batch SARAH achieve for general nonconvex objectives?
• RQ2 Under what conditions does SARAH enjoy linear convergence for gradient-dominated nonconvex functions?
• RQ3 How does mini-batch size affect convergence and complexity bounds for SARAH?
• RQ4 How does SARAH compare with SGD, SVRG, and GD in theory and practice for nonconvex empirical loss minimization?
• RQ5 What practical considerations arise for implementing SARAH and its variants (e.g., SARAH+) on neural networks?

主な発見

Method	Nonconvex	Tau-Gradient Dominated
GD	O(nL/ε)	O(nLτ log(1/ε))
SGD	O(Lσ^2/ε^2)	O(Lτσ^2/ε^2)
SVRG	O(n + n^{2/3}L/νε)	O((n + n^{2/3}Lτ/ν) log(1/ε))
SARAH	O(n + L^2/ε^2)	O((n + L^2τ^2) log(1/ε))

• SARAH-IN achieves sublinear convergence in expectation for general nonconvex P with appropriate η and inner loop length m
• For gradient-dominated (τ-gradient dominated) P, SARAH attains linear convergence to a global minimum under suitable η and m, with rates depending on τ and L
• The total IFO complexity to reach ε-accuracy is O(n + L^2/ε^2) in the general nonconvex setting, and O((n + L^2 τ^2) log(1/ε)) for gradient-dominated cases
• Mini-batch size b influences the allowable learning rate and inner loop size, with larger b enabling faster practical convergence
• The practical SARAH+ variant uses adaptive inner-loop termination and performs competitively against SVRG and SGD-based methods on neural networks (MNIST, CIFAR-10)