[Paper Review] On the Efficiency of Random Permutation for ADMM and Coordinate Descent

Ruoyu Sun, Zhi‐Quan Luo+1 more|arXiv (Cornell University)|Mar 22, 2015

Sparse and Compressive Sensing Techniques46 references14 citations

TL;DR

This paper establishes theoretical guarantees for randomly permuted ADMM (RP-ADMM) and block coordinate descent (RP-BCD), proving that RP-ADMM converges in expectation for solving linear systems and that RP-BCD achieves an expected convergence rate O(n) faster than cyclic BCD, with the spectrum of its update matrix confined to (−1/3, 1), offering a fundamental explanation for the empirical superiority of random permutation in large-scale optimization.

ABSTRACT

Random permutation is observed to be powerful for optimization algorithms: for multi-block ADMM (alternating direction method of multipliers), while the classical cyclic version divergence, the randomly permuted version converges in practice; for BCD (block coordinate descent), the randomly permuted version is typically faster than other versions. In this paper, we provide strong theoretical evidence that random permutation has positive effects on ADMM and BCD, by analyzing randomly permuted ADMM (RP-ADMM) for solving linear systems of equations, and randomly permuted BCD (RP-BCD) for solving unconstrained quadratic problems. First, we prove that RP-ADMM converges in expectation for solving systems of linear equations. The key technical result is that the spectrum of the expected update matrix of RP-BCD lies in $(-1/3, 1)$, instead of the typical range $(-1, 1)$. Second, we establish expected convergence rates of RP-ADMM for solving linear sytems and RP-BCD for solving unconstrained quadratic problems. This expected rate of RP-BCD is $O(n)$ times better than the worst-case rate of cyclic BCD, thus establishing a gap of at least $O(n)$ between RP-BCD and cyclic BCD. To analyze RP-BCD, we propose a conjecture of a new matrix AM-GM (algebraic mean-geometric mean) inequality, and prove a weaker version of it.

Motivation & Objective

• To resolve the divergence issue of cyclic multi-block ADMM in solving linear systems by analyzing randomly permuted ADMM (RP-ADMM).
• To establish a theoretical foundation for the empirical success of random permutation in block coordinate descent (BCD) and ADMM.
• To quantify the expected convergence rate of RP-BCD for unconstrained quadratic problems and compare it with cyclic BCD.
• To introduce and prove a weaker version of a new matrix AM-GM inequality to support the convergence analysis of RP-BCD.
• To demonstrate that random permutation yields a provable speedup over cyclic and independent randomization schemes in large-scale optimization.

Proposed method

• Analyzes RP-ADMM for solving non-singular square linear systems via the augmented Lagrangian framework.
• Derives the expected update matrix of RP-BCD and proves its spectrum lies in (−1/3, 1), a tighter bound than the typical (−1, 1).
• Establishes expected convergence rates for RP-ADMM on linear systems and RP-BCD on quadratic problems using spectral analysis.
• Proposes a conjecture on a new matrix AM-GM inequality and proves a weaker version to bound the convergence rate of RP-BCD.
• Employs mathematical induction and eigenvalue analysis to derive bounds on the expected residual decay over iterations.
• Validates theoretical findings with numerical experiments on various matrix types (Gaussian, log-normal, uniform, circulant Hankel).

Experimental results

Research questions

• RQ1 Does randomly permuted ADMM (RP-ADMM) converge in expectation for solving linear systems, despite the divergence of cyclic ADMM?
• RQ2 What is the expected convergence rate of RP-BCD for unconstrained quadratic problems, and how does it compare to cyclic BCD?
• RQ3 Can the spectrum of the update matrix for RP-BCD be bounded more tightly than (−1, 1), and what implications does this have for convergence?
• RQ4 Is there a provable speedup of RP-BCD over cyclic BCD, and if so, what is the order of magnitude of this gap?
• RQ5 Can a new matrix AM-GM inequality be formulated and partially proven to support the convergence analysis of randomized BCD?

Key findings

• RP-ADMM converges in expectation for solving non-singular linear systems, providing a theoretical remedy for the divergence of cyclic ADMM.
• The spectrum of the expected update matrix for RP-BCD lies in (−1/3, 1), a significant tightening from the standard (−1, 1) range.
• The expected convergence rate of RP-BCD is O(n) times faster than the worst-case rate of cyclic BCD, establishing a provable O(n) gap.
• Numerical experiments confirm that RP-ADMM converges faster than basic gradient descent and often more than 10 times faster than cyclic ADMM in divergent cases.
• For Gaussian and log-normal matrices, cyclic ADMM diverges with increasing dimension, while RP-ADMM consistently converges and outperforms gradient descent.
• BR-ADMM (Bernoulli-randomized ADMM) also converges but is generally slower than RP-ADMM, though it may save sampling time in constrained architectures.