[Paper Review] SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives

Aaron Defazio, Francis Bach+1 more|arXiv (Cornell University)|Jul 1, 2014

Sparse and Compressive Sensing Techniques17 references928 citations

TL;DR

SAGA is a novel incremental gradient method that achieves faster convergence than SAG and SVRG for both strongly convex and non-strongly convex composite problems, with support for proximal operators in the composite setting. It uses a variance-reduced update with a dynamic table of past gradients and automatically adapts to inherent strong convexity without requiring regularization or step size tuning.

ABSTRACT

In this work we introduce a new optimisation method called SAGA in the spirit of SAG, SDCA, MISO and SVRG, a set of recently proposed incremental gradient algorithms with fast linear convergence rates. SAGA improves on the theory behind SAG and SVRG, with better theoretical convergence rates, and has support for composite objectives where a proximal operator is used on the regulariser. Unlike SDCA, SAGA supports non-strongly convex problems directly, and is adaptive to any inherent strong convexity of the problem. We give experimental results showing the effectiveness of our method.

Motivation & Objective

• To develop a fast incremental gradient method that outperforms existing methods like SAG, SVRG, and SDCA in convergence speed and versatility.
• To support composite objectives with non-differentiable regularizers via proximal operators, extending applicability to L1-regularized problems.
• To enable direct application to non-strongly convex problems without adding artificial regularization, avoiding an extra hyperparameter.
• To achieve convergence rates that adapt to the actual strong convexity present in the problem, improving robustness and performance.

Proposed method

• SAGA maintains a table of past gradient values for each component function, enabling efficient variance reduction in the stochastic update.
• At each iteration, a random index j is selected, and the current gradient at x^k is used to update the table and compute a corrected gradient estimate.
• The update rule combines the current gradient, the previous gradient for the same component, and the average of all past gradients to form a low-variance estimate.
• A proximal operator is applied to the intermediate update to handle non-smooth regularizers like L1, enabling composite objective optimization.
• The method uses a step size of γ = 1/(2(μn + L)) in the strongly convex case and γ = 1/(3L) in the non-strongly convex case, with automatic adaptation to μ.
• The algorithm is designed to be simple and efficient, storing only one gradient per component, with minimal memory overhead.

Experimental results

Research questions

• RQ1 Can a new incremental gradient method be designed that achieves faster convergence than SAG and SVRG while supporting composite objectives?
• RQ2 Can the method handle non-strongly convex problems directly, without requiring artificial regularization?
• RQ3 Does the method automatically adapt to the actual level of strong convexity present in the problem, improving convergence in practice?
• RQ4 How does the theoretical convergence rate of SAGA compare to that of SDCA, SAG, and SVRG in both strongly and non-strongly convex settings?

Key findings

• SAGA achieves a linear convergence rate of (1 - μ/(2(μn + L)))^k in the strongly convex case, which is faster than SAG and SVRG and within a factor of 2 of SDCA.
• In the non-strongly convex case, SAGA achieves an O(1/k) convergence rate for the average iterate, with a step size of γ = 1/(3L), matching the best-known rates.
• The method automatically adapts to the actual strong convexity μ > 0 present in the problem, achieving a rate of (1 - min{1/(4n), μ/(3L)})^k without requiring μ as input.
• SAGA supports composite objectives via proximal operators, enabling efficient optimization for L1-regularized problems such as Lasso.
• Empirical results show SAGA performs competitively with Finito and SDCA, and outperforms SAG in early iterations, especially without adaptive step size tuning.
• The method avoids the need for a recalibration pass or additional hyperparameters, making it more practical than SVRG and SAG in non-strongly convex settings.