[Paper Review] On the Optimal Linear Convergence Rate of a Generalized Proximal Point Algorithm

Min Tao, Xiaoming Yuan|arXiv (Cornell University)|May 18, 2016

Optimization and Variational Analysis30 references3 citations

TL;DR

This paper establishes the optimal linear convergence rate for a generalized proximal point algorithm (PPA) under Rockafellar's condition, extending prior results to inexact and relaxed variants. It proves that the same condition ensuring linear convergence in classical PPA also guarantees optimal linear convergence for generalized PPA, including augmented Lagrangian and ADMM methods, with refined, weaker conditions specified for convex optimization contexts.

ABSTRACT

The proximal point algorithm (PPA) has been well studied in the literature. In particular, its linear convergence rate has been studied by Rockafellar in 1976 under certain condition. We consider a generalized PPA in the generic setting of finding a zero point of a maximal monotone operator, and show that the condition proposed by Rockafellar can also sufficiently ensure the linear convergence rate for this generalized PPA. Indeed we show that these linear convergence rates are optimal. Both the exact and inexact versions of this generalized PPA are discussed. The motivation to consider this generalized PPA is that it includes as special cases the relaxed versions of some splitting methods that are originated from PPA. Thus, linear convergence results of this generalized PPA can be used to better understand the convergence of some widely used algorithms in the literature. We focus on the particular convex minimization context and specify Rockafellar's condition to see how to ensure the linear convergence rate for some efficient numerical schemes, including the classical augmented Lagrangian method proposed by Hensen and Powell in 1969 and its relaxed version, the original alternating direction method of multipliers (ADMM) by Glowinski and Marrocco in 1975 and its relaxed version (i.e., the generalized ADMM by Eckstein and Bertsekas in 1992). Some refined conditions weaker than existing ones are proposed in these particular contexts.

Motivation & Objective

• To extend Rockafellar's linear convergence condition from classical PPA to a generalized PPA framework for finding zero points of maximal monotone operators.
• To establish linear convergence for both exact and inexact versions of the generalized PPA under the same condition as in the original PPA.
• To apply the generalized PPA to convex optimization problems, specifically the augmented Lagrangian method (ALM) and alternating direction method of multipliers (ADMM), and derive refined convergence conditions.
• To show that the convergence condition is optimal and weaker than existing conditions in the literature for ALM and ADMM variants.
• To unify the convergence analysis of major operator splitting methods, including ALM, ADMM, and their relaxed versions, under a single theoretical framework.

Proposed method

• Formulates a generalized PPA as $ z^{k+1} = J_{c_k T}(z^k) $ for maximal monotone operators $ T $, with $ c_k $ as proximal parameters.
• Analyzes both exact and inexact versions of the generalized PPA, where inexactness is controlled via relative error tolerances.
• Applies the condition from Rockafellar (1976) — that $ T^{-1} $ is Lipschitz continuous at 0 — to ensure linear convergence.
• Uses resolvent operator theory and metric subregularity to establish convergence rates, particularly focusing on the inverse of the operator $ S_{\lambda,\mathcal{A},\mathcal{B}} $.
• Derives specific conditions for linear convergence in convex minimization by specializing the general framework to ALM and ADMM.
• Employs the generalized ADMM as a special case of the generalized PPA, showing R-linear convergence of primal and dual iterates under weakened assumptions.

Experimental results

Research questions

• RQ1 Can Rockafellar’s linear convergence condition for classical PPA be extended to a generalized PPA framework with inexact subproblems?
• RQ2 What are the weakest sufficient conditions for linear convergence of the generalized PPA, particularly in convex optimization settings?
• RQ3 How do the convergence results for generalized PPA apply to widely used methods like ALM and ADMM?
• RQ4 Is the convergence condition in this paper optimal, and how does it compare to existing conditions in the literature?
• RQ5 Can the generalized PPA framework unify the convergence analysis of multiple splitting methods, including ADMM and its relaxed variants?

Key findings

• The generalized PPA with inexact subproblems achieves linear convergence under the same condition as classical PPA — that $ T^{-1} $ is Lipschitz continuous at 0.
• The linear convergence rate established is optimal, meaning no faster rate can be guaranteed under the same assumptions.
• For the augmented Lagrangian method (ALM), the paper specifies weaker conditions than previously known for linear convergence, particularly when $ f $ is strongly convex and $ \nabla f $ is Lipschitz continuous.
• For the generalized ADMM, the paper shows R-linear convergence of primal variables $ \{x^k\} $, dual variables $ \{p^k\} $, and $ \{Mx^k\} $, provided $ M $ is full column rank.
• The condition that $ S_{\lambda,\mathcal{A},\mathcal{B}}^{-1} $ is Lipschitz continuous at 0 is shown to be weaker than existing assumptions in the literature, such as those in [9] and [8].
• The framework unifies convergence analysis for ALM, ADMM, Douglas-Rachford, Peaceman-Rachford, and their generalized versions, providing a common theoretical basis for their linear convergence.