[论文解读] Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent

Xiangru Lian, Ce Zhang+4 more|arXiv (Cornell University)|May 25, 2017

Stochastic Gradient Optimization Techniques参考文献 40被引用 406

一句话总结

本论文分析 decentral...

ABSTRACT

Most distributed machine learning systems nowadays, including TensorFlow and CNTK, are built in a centralized fashion. One bottleneck of centralized algorithms lies on high communication cost on the central node. Motivated by this, we ask, can decentralized algorithms be faster than its centralized counterpart? Although decentralized PSGD (D-PSGD) algorithms have been studied by the control community, existing analysis and theory do not show any advantage over centralized PSGD (C-PSGD) algorithms, simply assuming the application scenario where only the decentralized network is available. In this paper, we study a D-PSGD algorithm and provide the first theoretical analysis that indicates a regime in which decentralized algorithms might outperform centralized algorithms for distributed stochastic gradient descent. This is because D-PSGD has comparable total computational complexities to C-PSGD but requires much less communication cost on the busiest node. We further conduct an empirical study to validate our theoretical analysis across multiple frameworks (CNTK and Torch), different network configurations, and computation platforms up to 112 GPUs. On network configurations with low bandwidth or high latency, D-PSGD can be up to one order of magnitude faster than its well-optimized centralized counterparts.

研究动机与目标

• Motivate and assess whether decentralized communication can beat centralized setups in distributed SGD.
• Provide theoretical analysis identifying regimes where D-PSGD matches or exceeds C-PSGD in total optimization effort.
• Empirically validate theory across frameworks (CNTK, Torch), network configurations, and large GPU clusters.
• Quantify practical speedups and communication patterns in real-world deployments.

提出的方法

• Propose Decentralized Parallel Stochastic Gradient Descent (D-PSGD) on a connected decentralized network.
• Model the problem as min_x f(x) with f(x)=1/n sum_i Eξ~Di Fi(x; ξ) and use a symmetric doubly stochastic weight matrix W to encode topology.
• Show update rule X_{k+1} = X_k W - γ ∂F(X_k; ξ_k) and analyze convergence without assuming bounded gradients or domains.
• Derive convergence bounds indicating similar rate to C-PSGD while reducing per-node communication to O(Deg(network)).
• Provide conditions under which linear speedup with more nodes is achievable in terms of iterations K and network spectral properties.
• Compare computational and communication complexities between C-PSGD and D-PSGD in Table 1.

实验结果

研究问题

• RQ1 Under what conditions can decentralized PSGD match or outperform centralized PSGD in terms of convergence and total computational effort?
• RQ2 How does the communication pattern and network topology affect the convergence and speedup of D-PSGD?
• RQ3 Can decentralized algorithms achieve linear speedup with increasing numbers of nodes, and what are the practical limits?
• RQ4 How do D-PSGD and C-PSGD compare empirically across frameworks, network configurations, and hardware scales?
• RQ5 What is the impact of network bandwidth and latency on the relative performance of decentralized vs centralized approaches?

主要发现

算法	通信复杂度	计算复杂度
C-PSGD (mini-batch SGD)	O(n)	O(n/ϵ + 1/ϵ^2)
D-PSGD	O(Deg(network))	O(n/ϵ + 1/ϵ^2)

• D-PSGD attains a convergence rate comparable to C-PSGD, with similar total computational complexity but reduced communication on the busiest node.
• As the number of nodes grows, D-PSGD can exhibit asymptotic linear speedup in computational effort (K large enough).
• On low-bandwidth or high-latency networks, D-PSGD can be up to about 10x faster than well-optimized centralized counterparts.
• Experiments across CNTK and Torch, networks up to 112 GPUs, show D-PSGD outperforming centralized methods under constrained communication.”
• The per-iteration communication cost for D-PSGD scales with network degree, unlike C-PSGD's O(n) bottleneck, enabling better scalability in sparse topologies (e.g., ring).
• Empirical results indicate similar training loss and accuracy trajectories between D-PSGD and centralized approaches, with D-PSGD achieving these with less communication overhead