[Paper Review] Convergence Rates of Inexact Proximal-Gradient Methods for Convex Optimization

Mark Schmidt, Nicolas Le Roux+1 more|arXiv (Cornell University)|Sep 12, 2011

Sparse and Compressive Sensing TechniquesEngineering39 references266 citations

TL;DR

This paper establishes that inexact proximal-gradient methods for convex optimization maintain the same convergence rates as exact methods when gradient and proximity operator errors decrease at appropriate rates. It proves that both basic and accelerated proximal-gradient algorithms achieve $O(1/k)$ and $O(1/k^2)$ convergence rates, respectively, under controlled error conditions, enabling efficient solution of non-smooth problems with expensive proximity operators.

ABSTRACT

We consider the problem of optimizing the sum of a smooth convex function and a non-smooth convex function using proximal-gradient methods, where an error is present in the calculation of the gradient of the smooth term or in the proximity operator with respect to the non-smooth term. We show that both the basic proximal-gradient method and the accelerated proximal-gradient method achieve the same convergence rate as in the error-free case, provided that the errors decrease at appropriate rates.Using these rates, we perform as well as or better than a carefully chosen fixed error level on a set of structured sparsity problems.

Motivation & Objective

• To analyze the convergence behavior of inexact proximal-gradient methods when errors are present in gradient or proximity operator computations.
• To establish conditions under which inexact methods achieve the same convergence rates as exact methods for convex and strongly convex problems.
• To demonstrate that controlled error decay enables performance comparable to or better than fixed error levels in structured sparsity problems.
• To provide theoretical justification for the empirical success of inexact methods in large-scale non-smooth optimization.

Proposed method

• Proposes inexact proximal-gradient methods where gradient and proximity operator computations include bounded errors $\|e_k\|$ and $\varepsilon_k$, respectively.
• Uses a Lyapunov function approach to bound the distance to the optimal solution $\|v_k - x^*\|$ via recursive inequalities involving error terms.
• Derives convergence bounds for both convex and strongly convex cases using the strong growth property $\|v_k - x^*\|^2 \leq \frac{2\delta_k}{\mu}$ and error decay rates.
• Introduces a key inequality (Eq. 21) that links error terms $\|e_k\|$ and $\varepsilon_k$ to the convergence rate via a geometric decay factor $\left(1 - \sqrt{\mu/L}\right)^k$.
• Applies Lemma 1 to bound the distance to optimality and derives a final convergence rate in terms of $\widehat{A}_k$, $\widehat{B}_k$, and initial error $\delta_0$.
• Establishes that the function value error $f(x_k) - f(x^*)$ decays at the same rate as in the exact case when $\|e_k\|$ and $\varepsilon_k$ decay sufficiently fast.

Experimental results

Research questions

• RQ1 Can inexact proximal-gradient methods preserve the same convergence rates as exact methods when errors are present in gradient or proximity operator computations?
• RQ2 What specific decay rates on gradient and proximity errors are sufficient to maintain $O(1/k)$ and $O(1/k^2)$ convergence for basic and accelerated methods, respectively?
• RQ3 How do error control strategies compare to fixed error levels in practice on structured sparsity problems?
• RQ4 Under what conditions does the inexact method achieve the same theoretical convergence rate as the exact method in both convex and strongly convex settings?
• RQ5 Can the theoretical error bounds be used to guide practical implementation of inexact proximal methods in non-smooth optimization?

Key findings

• The basic inexact proximal-gradient method achieves an $O(1/k)$ convergence rate for convex problems when the gradient and proximity errors decay appropriately.
• The accelerated inexact proximal-gradient method achieves an $O(1/k^2)$ convergence rate under the same error decay conditions, matching the optimal rate of exact accelerated methods.
• For strongly convex problems, the convergence rate is linear with a factor of $\left(1 - \sqrt{\mu/L}\right)^k$, provided that $\|e_k\|$ and $\varepsilon_k$ decay at a rate that keeps the error terms $\widehat{A}_k$ and $\widehat{B}_k$ bounded.
• The function value error satisfies $f(x_k) - f(x^*) \leq \left(1 - \sqrt{\mu/L}\right)^k \left( \sqrt{2(f(x_0) - f(x^*))} + \widehat{A}_k \sqrt{2/\mu} + \sqrt{\widehat{B}_k} \right)^2$, ensuring the same asymptotic rate as the exact method.
• Empirical results show that adaptive error control outperforms fixed error levels on structured sparsity problems, validating the theoretical findings.
• The analysis confirms that inexact methods can be as efficient as exact ones when errors are managed via decaying error tolerances, even for non-smooth regularizers like total variation and nuclear norm.