[论文解读] Multi-consensus Decentralized Accelerated Gradient Descent

Haishan Ye, Luo Luo+2 more|arXiv (Cornell University)|May 2, 2020

Distributed Control Multi-Agent Systems参考文献 33被引用 33

一句话总结

本文提出两种分布式加速近端梯度方法 ProxMudag 和 Mudag，在计算和通信复杂度方面达到（近似）最优，且依赖全局条件数，同时允许局部函数非凸。

ABSTRACT

This paper considers the decentralized convex optimization problem, which has a wide range of applications in large-scale machine learning, sensor networks, and control theory. We propose novel algorithms that achieve optimal computation complexity and near optimal communication complexity. Our theoretical results give affirmative answers to the open problem on whether there exists an algorithm that can achieve a communication complexity (nearly) matching the lower bound depending on the global condition number instead of the local one. Furthermore, the linear convergence of our algorithms only depends on the strong convexity of global objective and it does \emph{not} require the local functions to be convex. The design of our methods relies on a novel integration of well-known techniques including Nesterov's acceleration, multi-consensus and gradient-tracking. Empirical studies show the outperformance of our methods for machine learning applications.

研究动机与目标

• Address the decentralized convex optimization problem where the global objective combines local functions across agents.
• Develop algorithms that achieve (near) optimal communication complexity depending on the global condition number rather than the local one.
• Relax convexity requirements on local functions while preserving linear convergence under global strong convexity.
• Incorporate multi-consensus, gradient-tracking, and Nesterov acceleration to approximate centralized accelerated gradient methods.
• Provide empirical evidence of outperformance on machine learning tasks.

提出的方法

• Propose ProxMudag for composite objectives with a non-differentiable regularizer r(x); combines proximal updates, multi-consensus, and gradient tracking with acceleration.
• Propose Mudag for smooth objectives (r(x)=0); uses a two-step multi-consensus and gradient-tracking to emulate centralized accelerated gradient descent.
• Use FastMix as an efficient averaging operator to implement multi-consensus and ensure consensus among agents.
• Analyze convergence via a Lyapunov function and show averaged sequences follow proximal-accelerated gradient dynamics.
• Derive computation complexity O(sqrt(kappa_g) log(1/epsilon)) and near-optimal communication complexity O(sqrt(kappa_g/(1-lambda2(W))) log(M kappa_g / L) log(1/epsilon)).
• Provide iteration and communication bounds that depend on the global condition number kappa_g = L/μ rather than the local condition number.

实验结果

研究问题

• RQ1 Can decentralized optimization achieve near-optimal communication complexity that depends on the global condition number κ_g instead of the local κ_ℓ?
• RQ2 Is it possible to attain linear convergence for decentralized composite objectives without requiring each local function f_i to be convex?
• RQ3 Do accelerated proximal gradient strategies with multi-consensus and gradient-tracking suffice to approximate centralized accelerated methods in a decentralized setting?
• RQ4 What are the computation and communication complexity trade-offs for two proposed algorithms, ProxMudag and Mudag, under smoothness and strong convexity assumptions?

主要发现

Methods	Computation	Communication	Is f_i convex?
Mudag	O(√κ_g log(1/ε))	Ō(√(κ_g/(1−λ2(W))) log(Mκ_g/L) log(1/ε))	No
Lower Bound (Scaman et al., 2017)	Ω(√κ_g log(1/ε))	Ω(√(κ_g/(1−λ2(W))) log(1/ε))	N/A

• Mudag achieves computation complexity O(√κ_g log(1/ε)) and near-optimal communication complexity O(√(κ_g/(1−λ2(W))) log(Mκ_g/L) log(1/ε)).
• ProxMudag achieves optimal computation and near-optimal communication complexity for convex but non-differentiable r(x).
• The algorithms depend on the global condition number κ_g, not the local one, addressing an open question in decentralized optimization.
• The methods allow f_i to be non-convex while requiring global f to be μ-strongly convex and L-smooth, broadening applicability.
• FastMix and gradient-tracking are integrated to ensure averaging and gradient estimates approximate centralized accelerations.
• Empirical results show the proposed methods outperform existing decentralized methods on machine learning tasks.]
• table_headers:["Methods","Computation","Communication","Is f_i convex?"]
• table_rows:[["Mudag","O(√κ_g log(1/ε))","Ō(√(κ_g/(1−λ2(W))) log(Mκ_g/L) log(1/ε))","No"],["Lower Bound (Scaman et al., 2017)","Ω(√κ_g log(1/ε))","Ω(√(κ_g/(1−λ2(W))) log(1/epsilon))","N/A"]]}」} } ) } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } }
• table_headers_translated:[]} } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } }