[论文解读] A Survey of ADMM Variants for Distributed Optimization: Problems, Algorithms and Features

Yu Yang, Xiaohong Guan+4 more|arXiv (Cornell University)|Aug 7, 2022

Advanced MIMO Systems Optimization被引用 13

一句话总结

本文全面综述了用于分布式优化的ADMM变体，按问题类型——多块、耦合目标、非凸、一致性及异步设置——进行分类。分析了其算法设计、收敛性及实用特性，为智能电网、联邦学习和机器学习中的应用提供了教程与研究路线图。

ABSTRACT

By coordinating terminal smart devices or microprocessors to engage in cooperative computation to achieve systemlevel targets, distributed optimization is incrementally favored by both engineering and computer science. The well-known alternating direction method of multipliers (ADMM) has turned out to be one of the most popular tools for distributed optimization due to many advantages, such as modular structure, superior convergence, easy implementation and high flexibility. In the past decade, ADMM has experienced widespread developments. The developments manifest in both handling more general problems and enabling more effective implementation. Specifically, the method has been generalized to broad classes of problems (i.e.,multi-block, coupled objective, nonconvex, etc.). Besides, it has been extensively reinforced for more effective implementation, such as improved convergence rate, easier subproblems, higher computation efficiency, flexible communication, compatible with inaccurate information, robust to communication delays, etc. These developments lead to a plentiful of ADMM variants to be celebrated by broad areas ranging from smart grids, smart buildings, wireless communications, machine learning and beyond. However, there lacks a survey to document those developments and discern the results. To achieve such a goal, this paper provides a comprehensive survey on ADMM variants. Particularly, we discern the five major classes of problems that have been mostly concerned and discuss the related ADMM variants in terms of main ideas, main assumptions, convergence behaviors and main features. In addition, we figure out several important future research directions to be addressed. This survey is expected to work as a tutorial for both developing distributed optimization in broad areas and identifying existing theoretical research gaps.

研究动机与目标

• 系统性地对过去十年中为分布式优化开发的ADMM变体进行分类与分析。
• 识别并阐明文献中解决的五大类主要问题：多块、耦合目标、非凸、一致性及异步优化。
• 考察这些问题类别中ADMM变体的主要思想、假设、分解策略、收敛行为及关键特性。
• 突出其实际优势，如对通信延迟的鲁棒性、灵活的通信机制以及对不精确信息的兼容性。
• 识别关键的未来研究方向，包括非凸扩展、加速方法、异步实现，以及与强化学习和联邦学习的集成。

提出的方法

• 本综述根据五类核心问题类型对ADMM变体进行分类：多块、耦合目标、非凸、一致性及异步优化。
• 针对每一类问题，回顾其算法设计，包括分解策略及ADMM固有的对偶上升机制。
• 在各种假设下评估收敛性，如凸性、Lipschitz连续性及有界梯度。
• 分析包括通信灵活性、对延迟的鲁棒性以及对不精确子问题解的支持等特性。
• 讨论实现技术，如周期性通信、部分客户端参与及异步更新。
• 整合来自实际应用的案例研究——例如联邦学习、智能电网和无线通信——以说明其实际效用。

实验结果

研究问题

• RQ1 ADMM变体如何处理多块和耦合目标优化问题，其收敛性保证是什么？
• RQ2 哪些关键算法改进使ADMM能够在非凸设置下收敛，与经典ADMM有何不同？
• RQ3 ADMM变体在支持灵活通信模式（如间歇性或部分客户端参与）方面有何机制，这对收敛性有何影响？
• RQ4 ADMM如何适应更新频率和通信延迟变化的异步计算环境？
• RQ5 ADMM与联邦学习和多智能体强化学习等新兴范式之间存在哪些协同效应，尚存哪些挑战？

主要发现

• ADMM变体已被成功扩展以处理非凸问题，但其收敛性保证通常弱于凸情况。
• 若干ADMM变体通过过松弛、不精确求解及自适应惩罚参数等技术实现了更快的收敛速率。
• 采用周期性或部分客户端参与的ADMM可显著降低联邦学习中的通信开销，同时保持可接受的收敛性能。
• 异步ADMM变体对可变更新频率和通信延迟表现出强鲁棒性，适用于实时分布式系统。
• ADMM的模块化结构和低每轮迭代复杂度，使其与联邦学习等隐私保护机器学习框架高度兼容。
• 尽管性能优异，许多ADMM变体在一般非凸和随机设置下仍缺乏收敛性分析，表明存在关键的研究空白。