[Paper Review] General Convergence Rates Follow From Specialized Rates Assuming Growth Bounds

Benjamin Grimmer|arXiv (Cornell University)|May 15, 2019

Sparse and Compressive Sensing Techniques18 references4 citations

TL;DR

This paper introduces a meta-theorem that derives general convergence rates for first-order optimization methods from specialized rates assuming a Hölder growth bound (including quadratic and sharp growth). By leveraging the structure of existing convergence proofs under growth assumptions, the method lifts these rates to the general convex case without modifying the algorithm, enabling unified analysis across all settings with minimal overhead.

ABSTRACT

Often in the analysis of first-order methods, assuming the existence of a quadratic growth bound (a generalization of strong convexity) facilitates much stronger convergence analysis. Hence the analysis is done twice, once for the general case and once for the growth bounded case. We give a meta-theorem for deriving general convergence rates from those assuming a growth lower bound. Applying this simple but conceptually powerful tool to the proximal point method, the subgradient method, and the bundle method immediately recovers their known convergence rates for general convex optimization problems from their specialized rates. Future works studying first-order methods can assume growth bounds for the sake of analysis without hampering the generality of the results. Our results can be applied to lift any rate based on a Hölder growth bound. As a consequence, guarantees for minimizing sharp functions imply guarantees for both general functions and those satisfying quadratic growth.

Motivation & Objective

• Address the inefficiency of deriving separate convergence rates for general convex problems and those with growth bounds (e.g., quadratic or sharp growth).
• Eliminate the need for redundant analysis by showing that rates derived under growth assumptions can be systematically converted to general rates.
• Provide a general framework applicable to diverse first-order methods, including proximal point, subgradient, and bundle methods.
• Enable researchers to assume growth bounds during analysis without sacrificing generality in final convergence guarantees.
• Establish a theoretical foundation for lifting convergence rates based on Hölder growth bounds to the general convex case.

Proposed method

• Propose a meta-theorem that transforms convergence rates derived under a Hölder growth bound $ F(x) \geq F(x^*) + \alpha \|x - x^*\|^p $ into general rates for convex functions.
• Use the assumption that the method satisfies (A1)–(A3): dependence on function and subgradient values, bounded iterates, and convergence under growth conditions.
• Apply the lifting technique to three standard methods: proximal point, subgradient, and bundle methods, recovering known general rates from specialized ones.
• Derive bounds by analyzing the interplay between the growth parameter $\alpha$, the initial distance $\|x_0 - x^*\|$, and the desired accuracy $\epsilon$.
• Utilize logarithmic and inverse dependencies to control the number of iterations required for $\epsilon$-accuracy in the general case.
• Leverage known convergence rates under growth assumptions (e.g., from Du and Ruszczyński for the bundle method) and apply the meta-theorem to derive general rates.

Experimental results

Research questions

• RQ1 Can convergence rates derived under a growth bound be systematically converted to general convex convergence rates without modifying the algorithm?
• RQ2 To what extent do existing specialized convergence rates for first-order methods imply general rates under mild assumptions?
• RQ3 How can the analysis of complex methods like the bundle method be simplified by assuming growth bounds?
• RQ4 What is the minimal structural requirement on a first-order method for the lifting technique to be applicable?
• RQ5 Can the lifting framework be generalized beyond quadratic and sharp growth to arbitrary Hölder growth bounds?

Key findings

• The general convergence rate for the proximal point method is $ O(\|x_0 - x^*\|^2 / \epsilon) $, recovered from its specialized rate under quadratic growth.
• The subgradient method achieves a general rate of $ O(\|x_0 - x^*\|^2 / \epsilon^2) $, matching known bounds, derived from its sharp growth rate.
• For the bundle method, the general rate is $ O(\|x_0 - x^*\|^4 / \epsilon^3) $, matching Kiwiel’s result up to logarithmic factors.
• The lifting technique applies uniformly across methods and does not require algorithmic changes, preserving the original method’s structure.
• The derived general rates are tight and match established bounds, confirming the method’s correctness and practical utility.
• The framework allows researchers to assume growth bounds during analysis while still obtaining valid, general convergence guarantees.