[Paper Review] Global Convergence of Unmodified 3-Block ADMM for a Class of Convex Minimization Problems

Tianyi Lin, Shiqian Ma+1 more|arXiv (Cornell University)|May 16, 2015

Sparse and Compressive Sensing Techniques48 references6 citations

TL;DR

This paper establishes global convergence of the unmodified 3-block ADMM for a class of convex minimization problems where the third block's objective function is smooth and strongly convex with a condition number in [1, 1.0798), under mild additional assumptions. The key contribution is proving that the 3-block ADMM converges globally for any penalty parameter γ > 0, making it parameter-unrestricted—unlike prior results that required γ to be bounded above, which limited practical efficiency.

ABSTRACT

The alternating direction method of multipliers (ADMM) has been successfully applied to solve structured convex optimization problems due to its superior practical performance. The convergence properties of the 2-block ADMM have been studied extensively in the literature. Specifically, it has been proven that the 2-block ADMM globally converges for any penalty parameter $γ>0$. In this sense, the 2-block ADMM allows the parameter to be free, i.e., there is no need to restrict the value for the parameter when implementing this algorithm in order to ensure convergence. However, for the 3-block ADMM, Chen \etal \cite{Chen-admm-failure-2013} recently constructed a counter-example showing that it can diverge if no further condition is imposed. The existing results on studying further sufficient conditions on guaranteeing the convergence of the 3-block ADMM usually require $γ$ to be smaller than a certain bound, which is usually either difficult to compute or too small to make it a practical algorithm. In this paper, we show that the 3-block ADMM still globally converges with any penalty parameter $γ>0$ if the third function $f_3$ in the objective is smooth and strongly convex, and its condition number is in $[1,1.0798)$, besides some other mild conditions. This requirement covers an important class of problems to be called regularized least squares decomposition (RLSD) in this paper.

Motivation & Objective

• To resolve the open problem of whether the 3-block ADMM can globally converge without restricting the penalty parameter γ, given that it can diverge without additional conditions.
• To identify sufficient conditions under which the 3-block ADMM remains globally convergent for any γ > 0, thus achieving parameter-unrestricted convergence.
• To extend the convergence guarantees of 2-block ADMM—known to converge globally for any γ > 0—to the 3-block case under specific structural assumptions on the third function f₃.
• To identify and analyze a natural class of problems, termed Regularized Least Squares Decomposition (RLSD), where the 3-block ADMM is globally convergent with any γ > 0.

Proposed method

• The authors analyze the 3-block ADMM for a structured convex optimization problem with three blocks, where the third function f₃ is smooth and strongly convex with condition number in [1, 1.0798).
• They derive convergence guarantees using a Lyapunov function approach, showing that the augmented Lagrangian decreases sufficiently at each iteration under the stated conditions.
• The method relies on analyzing the descent property of the augmented Lagrangian and proving that the sequence of iterates remains bounded and converges to a solution.
• The proof technique extends previous convergence frameworks for ADMM by incorporating the strong convexity and smoothness of f₃ to control the dual update and ensure global convergence.
• The analysis assumes A₃ = I and removes the variable x₃ from the constraint set, simplifying the structure to focus on f₃’s role in stabilization.
• The authors use a novel potential function that combines the augmented Lagrangian and a quadratic term related to f₃’s strong convexity to prove convergence.

Experimental results

Research questions

• RQ1 Can the 3-block ADMM be globally convergent for any penalty parameter γ > 0, without requiring γ to be bounded above?
• RQ2 What structural conditions on the third block’s objective function f₃ ensure global convergence of the 3-block ADMM with arbitrary γ > 0?
• RQ3 Does the condition number of f₃ play a critical role in enabling parameter-unrestricted convergence of the 3-block ADMM?
• RQ4 Is there a natural class of problems for which the 3-block ADMM is globally convergent with any γ > 0, making it practically efficient?

Key findings

• The 3-block ADMM globally converges for any γ > 0 when f₃ is smooth and strongly convex with condition number in [1, 1.0798), under mild assumptions.
• This result establishes the first global convergence guarantee for unmodified 3-block ADMM with no upper bound on γ, resolving a key limitation of prior work.
• The convergence is proven under the condition that A₃ = I and x₃ is not constrained, focusing the analysis on the role of f₃’s strong convexity and smoothness.
• The method is applicable to a broad class of problems called Regularized Least Squares Decomposition (RLSD), which includes many real-world applications in machine learning and signal processing.
• Numerical experiments confirm that the 3-block ADMM with γ = 0.7 and γ = 1.2 converges reliably across different problem sizes, with iteration counts and CPU times increasing moderately.
• In contrast, the 2-block ADMM variant diverges or stagnates for larger problems when γ is increased beyond a threshold, highlighting the importance of the proposed condition on f₃.