[Paper Review] ADMM Penalty Parameter Selection by Residual Balancing

Brendt Wohlberg|arXiv (Cornell University)|Apr 20, 2017

Sparse and Compressive Sensing Techniques18 references52 citations

TL;DR

The paper analyzes adaptive penalty parameter schemes for ADMM, shows a flaw in a popular residual-balancing heuristic, and proposes a mitigation using relative residuals and scaling considerations.

ABSTRACT

Appropriate selection of the penalty parameter is crucial to obtaining good performance from the Alternating Direction Method of Multipliers (ADMM). While analytic results for optimal selection of this parameter are very limited, there is a heuristic method that appears to be relatively successful in a number of different problems. The contribution of this paper is to demonstrate that their is a potentially serious flaw in this heuristic approach, and to propose a modification that at least partially addresses it.

Motivation & Objective

• Assess the effectiveness of existing adaptive penalty parameter rules for ADMM.
• Identify scaling-related issues that affect penalty updates.
• Propose a modification to improve robustness of penalty adaptation across problem scales.
• Illustrate concepts with Basis Pursuit DeNoising (BPDN) and CBPDN examples.

Proposed method

• Review ADMM formulation and residual-based convergence metrics.
• Analyze scaling properties of ADMM under problem transformations.
• Examine He et al.'s residual-balancing update rule and its limitations.
• Propose a scale-invariant update using relative residuals (r_rel and s_rel).
• Discuss adaptive multiplier policies and potential enhancements (tau adjustments, local scaling via xi).
• Apply concepts to BPDN and CBPDN to illustrate behavior.

Experimental results

Research questions

• RQ1 Does the standard residual-balancing adaptive rule for the ADMM penalty parameter reliably drive convergence across scaled problem instances?
• RQ2 How do problem scaling and residual definitions affect the effectiveness and invariance of adaptive penalty updates?
• RQ3 Can residual balancing be reformulated to be scaling-invariant without sacrificing convergence guarantees?
• RQ4 What practical modifications (e.g., relative residuals, adaptive tau, scaled updates) improve robustness in sparse representation problems?

Key findings

• The paper demonstrates a potentially serious flaw in the common residual-balancing heuristic for ADMM penalty parameter updates.
• It shows that the residuals used in the standard update do not scale consistently under problem scaling, potentially degrading performance when problem formulations are scaled.
• A proposed modification uses relative residuals to achieve scaling invariance in the penalty update, addressing the identified flaw.
• The convergence framework supports using relative residuals with adjusted stopping criteria while preserving convergence guarantees.
• The discussion includes adaptive multiplier policies and practical considerations (e.g., updating frequency, tau) to balance computational cost and convergence speed.
• Illustrative examples include Basis Pursuit DeNoising and Convolutional BPDN to highlight the impact on iteration count and residual evolution.