[Paper Review] A Universal Catalyst for First-Order Optimization

Hongzhou Lin, Julien Mairal+1 more|arXiv (Cornell University)|Jun 6, 2015

Stochastic Gradient Optimization TechniquesComputer Science30 references185 citations

TL;DR

This paper introduces a universal catalyst framework that accelerates a broad class of first-order optimization methods—including gradient descent, SAG, SAGA, MISO, and SVRG—by embedding them within an inexact proximal point algorithm with adaptive regularization. The method achieves optimal convergence rates for both convex and non-strongly convex problems without requiring prior knowledge of strong convexity parameters, significantly improving performance on ill-conditioned problems through theoretical and empirical acceleration.

ABSTRACT

We introduce a generic scheme for accelerating first-order optimization methods in the sense of Nesterov, which builds upon a new analysis of the accelerated proximal point algorithm. Our approach consists of minimizing a convex objective by approximately solving a sequence of well-chosen auxiliary problems, leading to faster convergence. This strategy applies to a large class of algorithms, including gradient descent, block coordinate descent, SAG, SAGA, SDCA, SVRG, Finito/MISO, and their proximal variants. For all of these methods, we provide acceleration and explicit support for non-strongly convex objectives. In addition to theoretical speed-up, we also show that acceleration is useful in practice, especially for ill-conditioned problems where we measure significant improvements.

Motivation & Objective

• To develop a generic acceleration framework applicable to a wide range of first-order methods in machine learning and signal processing.
• To address the open problem of accelerating incremental methods like SAG, SAGA, MISO, and SVRG beyond strongly convex settings.
• To eliminate the need for manual regularization parameter selection in non-strongly convex problems by enabling direct support for such objectives.
• To extend proximal variants of MISO to handle composite objectives and remove the big data condition $ n \geq \beta L/\mu $.

Proposed method

• The catalyst framework embeds any first-order method within a sequence of auxiliary subproblems using a proximal point-like regularization with dynamically adjusted parameters.
• It employs an inexact accelerated proximal point algorithm with a novel inexactness criterion that balances accuracy and computational cost.
• The method uses a momentum-based update strategy inspired by Nesterov’s acceleration, adapted to work with inexact subproblem solutions.
• A key component is the adaptive choice of the regularization parameter $ \kappa $, which depends on problem condition number and data size $ n $, enabling acceleration when $ n $ is large.
• The algorithm maintains primal iterates and uses a restarting mechanism to ensure convergence, with inner iterations solving subproblems using standard methods like SAG or MISO.
• It provides a primal-only convergence analysis, avoiding reliance on duality, and introduces a practical optimality certificate that does not require evaluating the dual objective.

Experimental results

Research questions

• RQ1 Can a universal acceleration framework be designed to work with a broad class of first-order methods, including incremental and proximal variants?
• RQ2 Does the catalyst approach achieve optimal convergence rates for both convex and non-strongly convex objectives without requiring prior knowledge of strong convexity?
• RQ3 Can acceleration be successfully applied to incremental methods like SAG, SAGA, MISO, and SVRG, which previously lacked known accelerated variants?
• RQ4 How can the big data condition $ n \geq \beta L/\mu $ be removed in proximal MISO while preserving convergence and acceleration?
• RQ5 What is the theoretical and empirical impact of the catalyst on ill-conditioned problems commonly found in machine learning?

Key findings

• The catalyst achieves an iteration complexity of $ O\left(\min\left\{\frac{L}{\mu},\sqrt{\frac{nL}{\mu}}\right\} \log\left(\frac{1}{\varepsilon}\right)\log\left(\frac{L}{\mu}\right)\right) $ for $ \mu $-strongly convex objectives, matching optimal rates.
• For non-strongly convex problems, the method provides explicit acceleration without requiring manual regularization, eliminating the need to pre-select $ \varepsilon $ in $ \varepsilon \|x\|^2 $-regularization.
• Empirical results show significant performance improvements on ill-conditioned problems, especially for SAG, SAGA, and MISO-Prox, with faster convergence in practice.
• The accelerated MISO-Prox variant removes the big data condition $ n \geq \beta L/\mu $, enabling broader applicability.
• The method achieves optimal $ O(1/k^2) $ convergence rate for convex problems and $ O((1 - \sqrt{\mu/L})^k) $ for strongly convex ones, regardless of the base method.
• The implementation uses theory-driven parameter settings for $ \kappa $, $ \alpha_0 $, and $ \varepsilon_k $, with $ \varepsilon_k $ decaying as $ (1-\rho)^k $ or $ 1/(k+2)^{4+\eta} $, ensuring convergence.