[Paper Review] RECAPP: Crafting a More Efficient Catalyst for Convex Optimization

Yair Carmon, Arun Jambulapati+2 more|arXiv (Cornell University)|Jun 17, 2022

Stochastic Gradient Optimization Techniques4 citations

TL;DR

This paper introduces RECAPP, a relaxed error criterion for accelerated proximal point methods that eliminates the need for high-accuracy subproblem solutions in convex optimization. By replacing the standard accuracy requirement with a looser condition, RECAPP achieves optimal complexity bounds—removing a logarithmic factor—on finite-sum and max-structured problems, matching or improving upon existing Catalyst-based and problem-specific methods.

ABSTRACT

The accelerated proximal point algorithm (APPA), also known as "Catalyst", is a well-established reduction from convex optimization to approximate proximal point computation (i.e., regularized minimization). This reduction is conceptually elegant and yields strong convergence rate guarantees. However, these rates feature an extraneous logarithmic term arising from the need to compute each proximal point to high accuracy. In this work, we propose a novel Relaxed Error Criterion for Accelerated Proximal Point (RECAPP) that eliminates the need for high accuracy subproblem solutions. We apply RECAPP to two canonical problems: finite-sum and max-structured minimization. For finite-sum problems, we match the best known complexity, previously obtained by carefully-designed problem-specific algorithms. For minimizing $\max_y f(x,y)$ where $f$ is convex in $x$ and strongly-concave in $y$, we improve on the best known (Catalyst-based) bound by a logarithmic factor.

Motivation & Objective

• To address the theoretical and practical inefficiency of existing Catalyst/accelerated proximal point methods, which require high-accuracy subproblem solutions.
• To eliminate the extraneous logarithmic factor in convergence rates that arises from stringent accuracy requirements in proximal point computations.
• To develop a general-purpose framework that maintains acceleration benefits while reducing computational overhead from subproblem solves.
• To achieve optimal or near-optimal complexity bounds for finite-sum and max-structured minimization problems without relying on problem-specific designs.

Proposed method

• Propose a new Relaxed Error Condition (RECAPP) that allows subproblem solutions to be computed with lower accuracy, reducing the number of required iterations.
• Introduce a novel error bound that enables the use of standard subproblem solvers (e.g., SVRG, mirror-prox) without incurring additional logarithmic terms in complexity.
• Apply RECAPP to finite-sum problems by combining it with SVRG, achieving the best-known complexity bound of $ O(n\log\log n + \sqrt{nLR^2/\epsilon}) $.
• Extend RECAPP to max-structured problems by using mirror-prox as the subproblem solver, achieving $ O(LR/\sqrt{\mu\epsilon}) $ gradient queries, removing a logarithmic factor from prior Catalyst-based bounds.
• Use a restarting strategy to achieve linear convergence for strongly convex objectives.
• Formalize the complexity analysis using a new variant of the approximate proximal oracle with error tolerance, ensuring convergence without extra logarithmic terms.

Experimental results

Research questions

• RQ1 Can the logarithmic factor in Catalyst-based convergence rates be removed without sacrificing generality or stability in convex optimization?
• RQ2 Is it possible to design a relaxed error condition for proximal point methods that maintains acceleration while reducing subproblem accuracy requirements?
• RQ3 Can RECAPP achieve optimal complexity bounds for finite-sum problems without relying on problem-specific algorithmic designs?
• RQ4 Does RECAPP improve the state-of-the-art complexity for min-max problems with convex-concave objectives?
• RQ5 Can the framework be extended to handle both non-strongly and strongly convex settings with optimal convergence rates?

Key findings

• For finite-sum problems, RECAPP combined with SVRG achieves the best-known complexity bound of $ O(n\log\log n + \sqrt{nLR^2/\epsilon}) $, matching the optimal rate of specialized algorithms.
• For minimizing $ \max_y f(x,y) $ with $ f $ convex in $ x $ and $ \mu $-strongly concave in $ y $, RECAPP removes a logarithmic factor, achieving $ O(LR/\sqrt{\mu\epsilon}) $ gradient queries, which is asymptotically optimal.
• The framework improves upon the prior Catalyst-based bound of $ O(LR/\sqrt{\mu\epsilon} \log(LR/\sqrt{\mu\epsilon})) $ by eliminating the logarithmic term.
• The method maintains global convergence guarantees while reducing the number of expensive subproblem solves, improving practical efficiency.
• Preliminary experiments on logistic regression show RECAPP is competitive with Catalyst-SVRG in practice.
• Theoretical analysis confirms that RECAPP enables optimal complexity in both non-strongly and strongly convex settings, with tight bounds on gradient and subproblem query complexity.