[论文解读] On the o(1/k) Convergence and Parallelization of the Alternating Direction Method of Multipliers

Wei Deng, Ming‐Jun Lai+1 more|arXiv (Cornell University)|Dec 11, 2013

Sparse and Compressive Sensing Techniques被引用 22

一句话总结

本文提出了一种用于大规模凸优化的并行与分布式ADMM变体，将问题分解为N个子问题并行求解。通过引入近端项和动态参数调节，建立了全局o(1/k)收敛性，显著提升了在Amazon EC2等分布式系统上的实际性能。

ABSTRACT

This paper introduces a parallel and distributed extension to the alternating direction method of multipliers (ADMM) for solving convex problem: minimize $\sum_{i=1}^N f_i(x_i)$ subject to $\sum_{i=1}^N A_i x_i=c, x_i\in \mathcal{X}_i$. The algorithm decomposes the original problem into N smaller subproblems and solves them in parallel at each iteration. This Jacobian-type algorithm is well suited for distributed computing and is particularly attractive for solving certain large-scale problems. This paper introduces a few novel results. Firstly, it shows that extending ADMM straightforwardly from the classic Gauss-Seidel setting to the Jacobian setting, from 2 blocks to N blocks, will preserve convergence if matrices $A_i$ are mutually near-orthogonal and have full column-rank. Secondly, for general matrices $A_i$, this paper proposes to add proximal terms of different kinds to the N subproblems so that the subproblems can be solved in flexible and efficient ways and the algorithm converges globally at a rate of o(1/k). Thirdly, a simple technique is introduced to improve some existing convergence rates from O(1/k) to o(1/k). In practice, some conditions in our convergence theorems are conservative. Therefore, we introduce a strategy for dynamically tuning the parameters in the algorithm, leading to substantial acceleration of the convergence in practice. Numerical results are presented to demonstrate the efficiency of the proposed method in comparison with several existing parallel algorithms. We implemented our algorithm on Amazon EC2, an on-demand public computing cloud, and report its performance on very large-scale basis pursuit problems with distributed data.

研究动机与目标

• 将ADMM从顺序的Gauss-Seidel更新扩展为N个块上的并行雅可比更新，以支持分布式计算。
• 通过在矩阵A_i不近似正交时引入近端项，放宽条件以确保全局收敛。
• 通过一种新颖的参数调节策略，将收敛速率从O(1/k)提升至o(1/k)。
• 利用基于云的分布式系统，实现大规模基追踪问题的高效求解。

提出的方法

• 通过在N个块上分割变量和约束，将问题分解为N个子问题。
• 采用雅可比型ADMM更新，使所有子问题在每次迭代中并行求解。
• 在子问题中引入具有不同结构的近端项，以确保对一般A_i矩阵的收敛性。
• 采用动态参数调节策略以在实际中加速收敛，降低理论边界的保守性。
• 在Amazon EC2上实现该算法，评估其在具有分布式数据的大规模基追踪问题上的性能。
• 将该方法应用于分布式基追踪，在云基础设施上展示了其可扩展性和效率。

实验结果

研究问题

• RQ1 能否将ADMM从2块的Gauss-Seidel更新扩展为N块的雅可比更新，同时保持收敛性？
• RQ2 在N块雅可比ADMM设置下，A_i矩阵需满足何种条件才能保证全局收敛？
• RQ3 如何设计近端项，以实现灵活且高效的子问题求解，并达到o(1/k)收敛？
• RQ4 能否通过算法改进将收敛速率从O(1/k)提升至o(1/k)？
• RQ5 动态参数调节在实际中是否能显著加速收敛？

主要发现

• 当在子问题中加入近端项时，所提出的ADMM变体即使在一般A_i矩阵下，也能实现全局o(1/k)收敛速率。
• 当A_i矩阵相互近似正交且列满秩时，雅可比更新方案仍能保持收敛性。
• 一个简单修改即可将现有收敛速率从O(1/k)提升至o(1/k)，从而增强理论保证。
• 动态参数调节在实际中显著加速了收敛，减少了对保守理论边界的依赖。
• 在Amazon EC2上的数值结果表明，与现有并行ADMM算法相比，该方法在大规模基追踪问题上表现出更优性能。
• 该方法在分布式数据上具有高效的可扩展性，展示了在现实世界大规模优化中的实际可行性。