[Paper Review] A convergence proof of the split Bregman method for regularized least-squares problems

Hung Nien, Jeffrey A. Fessler|arXiv (Cornell University)|Feb 18, 2014

Sparse and Compressive Sensing TechniquesEngineering17 citations

TL;DR

This paper proves the convergence of the split Bregman (SB) method for regularized least-squares problems with inexact updates, showing it is equivalent to a convergent alternating direction method of multipliers (ADMM) when the data-fitting term is quadratic. The key contribution is a simple convergence proof under practical conditions—without requiring full column rank or exact subproblem solutions—while demonstrating that ADMM with optimized parameters can outperform SB in convergence speed.

ABSTRACT

The split Bregman (SB) method [T. Goldstein and S. Osher, SIAM J. Imaging Sci., 2 (2009), pp. 323-43] is a fast splitting-based algorithm that solves image reconstruction problems with general l1, e.g., total-variation (TV) and compressed sensing (CS), regularizations by introducing a single variable split to decouple the data-fitting term and the regularization term, yielding simple subproblems that are separable (or partially separable) and easy to minimize. Several convergence proofs have been proposed, and these proofs either impose a "full column rank" assumption to the split or assume exact updates in all subproblems. However, these assumptions are impractical in many applications such as the X-ray computed tomography (CT) image reconstructions, where the inner least-squares problem usually cannot be solved efficiently due to the highly shift-variant Hessian. In this paper, we show that when the data-fitting term is quadratic, the SB method is a convergent alternating direction method of multipliers (ADMM), and a straightforward convergence proof with inexact updates is given using [J. Eckstein and D. P. Bertsekas, Mathematical Programming, 55 (1992), pp. 293-318, Theorem 8]. Furthermore, since the SB method is just a special case of an ADMM algorithm, it seems likely that the ADMM algorithm will be faster than the SB method if the augmented Largangian (AL) penalty parameters are selected appropriately. To have a concrete example, we conduct a convergence rate analysis of the ADMM algorithm using two splits for image restoration problems with quadratic data-fitting term and regularization term. According to our analysis, we can show that the two-split ADMM algorithm can be faster than the SB method if the AL penalty parameter of the SB method is suboptimal. Numerical experiments were conducted to verify our analysis.

Motivation & Objective

• To establish a convergence proof for the split Bregman method under practical conditions where full column rank and exact updates are not satisfied.
• To show that the split Bregman method is equivalent to an alternating direction method of multipliers (ADMM) when the data-fitting term is quadratic.
• To analyze the convergence rate of ADMM with two split variables and compare it to the SB method.
• To provide insight into parameter tuning for ADMM and SB by identifying conditions under which ADMM converges faster than SB.
• To validate the theoretical analysis with numerical experiments on image restoration problems using inexact solvers.

Proposed method

• Prove that the split Bregman method is equivalent to a two-split ADMM algorithm for regularized least-squares problems with quadratic data-fitting terms.
• Use Theorem 8 from [8] to establish convergence of the ADMM formulation, which implies convergence of the SB method under inexact updates.
• Formulate the ADMM subproblems: an x-update solving a least-squares problem with an augmented Lagrangian term, and a v-update solving a proximal operator of the regularizer.
• Analyze the convergence rate of the two-split ADMM algorithm using spectral analysis, deriving conditions under which it outperforms the SB method.
• Conduct numerical experiments using PCG with a circulant preconditioner for inexact x-updates in image restoration, comparing convergence speed across different penalty parameters.
• Use a masked finite difference matrix and a quadratic roughness penalty to model edge-preserving regularization, with parameter tuning based on the theoretical analysis.

Experimental results

Research questions

• RQ1 Can the split Bregman method be proven convergent when the system matrix does not have full column rank and subproblems are solved inexactly?
• RQ2 Is the split Bregman method equivalent to an ADMM algorithm under quadratic data-fitting terms, and does this equivalence enable a simpler convergence proof?
• RQ3 Under what conditions can ADMM with two split variables converge faster than the split Bregman method?
• RQ4 How does the choice of the augmented Lagrangian penalty parameter affect the convergence rate of both SB and ADMM algorithms?
• RQ5 Can theoretical convergence rate analysis predict the performance of inexact ADMM and SB solvers in image reconstruction with non-FFT solvers?

Key findings

• The split Bregman method is equivalent to a convergent ADMM algorithm when the data-fitting term is quadratic, enabling convergence proofs with inexact updates and without requiring full column rank.
• The convergence proof holds under general convex regularizers and allows inexact solutions to the x-update subproblem, which is critical for applications like parallel MR and CT imaging.
• ADMM with two split variables can converge faster than the split Bregman method when the AL penalty parameter of SB is suboptimal, particularly when it is under-estimated.
• Numerical experiments confirm that when the penalty parameter η is under-estimated (e.g., η = α/20), the ADMM algorithm converges significantly faster than SB, despite similar asymptotic rates.
• The ADMM algorithm with ρ = η/α = 1/20 exhibits strong overshoots and oscillations due to a large step size, highlighting the sensitivity of convergence to parameter tuning.
• With optimal parameter choice (e.g., η = α), the two-split ADMM achieves the fastest convergence with no ripple, and the convergence rate is only slightly degraded by inexact x-updates due to effective preconditioning.