[Paper Review] Weighted iteration complexity of the sPADMM on the KKT residuals for convex composite optimization

Li Shen, Shaohua Pan|arXiv (Cornell University)|Nov 10, 2016

Sparse and Compressive Sensing Techniques25 references5 citations

TL;DR

This paper establishes an O(1/k) weighted iteration complexity for the sPADMM on KKT residuals in convex composite optimization, using a novel generalized HPE framework. It closes a critical gap by proving ergodic convergence rates for ADMM with large step-sizes and proximal variants, ensuring global convergence with explicit suboptimality bounds.

ABSTRACT

In this paper we establish an $\mathcal{O}({1}/{k})$ weighted iteration complexity on the KKT residuals yielded by the sPADMM (semi-proximal alternating direction method of multiplier) for the convex composite optimization problem. This result, which is derived with the help of a novel generalized HPE (hybrid proximal extra-gradient) iteration formula, first fills the gap on the ergodic iteration complexity of the classic ADMM with a large step-size and its many proximal variants.

Motivation & Objective

• Address the lack of ergodic iteration complexity analysis for the classic ADMM and its proximal variants when using large step-sizes (≥ golden ratio).
• Provide a convergence rate guarantee for the sPADMM on KKT residuals in convex composite optimization problems.
• Develop a novel generalized HPE iteration formula to derive weighted iteration complexity bounds.
• Establish global convergence with explicit suboptimality bounds for the sPADMM under minimal assumptions.
• Bridge theoretical gaps in the convergence analysis of ADMM-type methods for multi-block and non-smooth optimization problems.

Proposed method

• Introduce a generalized HPE framework tailored to the sPADMM, enabling analysis of weighted iteration complexity.
• Define a new Lyapunov function based on the primal-dual iterate difference and proximal terms to track convergence.
• Use a three-term splitting structure in the sPADMM iteration to decouple the update of y, z, and x variables.
• Establish recursive inequalities involving the distance to the solution set T⁻¹(0) and the norm of the primal residual.
• Derive upper bounds for the initial progress terms d₁ and η₁ using the distance from the initial iterate to the solution set.
• Employ a weighted norm structure with operators S, T, and Σ to control the convergence rate and ensure boundedness.

Experimental results

Research questions

• RQ1 Can an O(1/k) weighted iteration complexity be established for the sPADMM on KKT residuals in convex composite optimization?
• RQ2 Does the sPADMM achieve ergodic convergence with explicit suboptimality bounds when the step-size exceeds the golden ratio?
• RQ3 Can a generalized HPE framework be constructed to unify and extend convergence analysis for ADMM and its proximal variants?
• RQ4 How do the initial iterate and problem structure affect the convergence rate of the sPADMM?
• RQ5 Is it possible to derive iteration complexity bounds without requiring strong positive definiteness of the proximal terms?

Key findings

• The sPADMM achieves an O(1/k) weighted iteration complexity on the KKT residuals for convex composite optimization problems.
• The convergence rate is established using a novel generalized HPE framework, which enables analysis under minimal assumptions.
• The result fills a critical gap in the ergodic iteration complexity literature for ADMM with large step-sizes (≥ golden ratio).
• The upper bounds for d₁ and η₁ are proportional to the squared distance from the initial iterate to the solution set T⁻¹(0).
• The analysis does not require the proximal terms S and T to make S + Σ_φ_f + βAA* or T + Σ_φ_g + βBB* positive definite.
• The convergence is global and suboptimality is quantified via a weighted norm involving the KKT residual and primal-dual iterates.