[论文解读] Communication-efficient SGD: From Local SGD to One-Shot Averaging

Artin Spiridonoff, Alex Olshevsky+1 more|arXiv (Cornell University)|Jun 9, 2021

Stochastic Gradient Optimization Techniques参考文献 41被引用 8

一句话总结

本文提出了一种通信高效的局部SGD变体，仅需$\Omega(N)$次通信即可实现最优收敛速率$1/(NT)$，且该通信次数与迭代次数$T$无关。结果表明，在温和的光滑性假设下，单次平均（仅一次通信回合）也能在渐近意义上实现线性加速，相比以往通信量随$\text{poly}(\log T)$或$\sqrt{T}$增长的方法，显著降低了通信开销。

ABSTRACT

We consider speeding up stochastic gradient descent (SGD) by parallelizing it across multiple workers. We assume the same data set is shared among $N$ workers, who can take SGD steps and coordinate with a central server. While it is possible to obtain a linear reduction in the variance by averaging all the stochastic gradients at every step, this requires a lot of communication between the workers and the server, which can dramatically reduce the gains from parallelism. The Local SGD method, proposed and analyzed in the earlier literature, suggests machines should make many local steps between such communications. While the initial analysis of Local SGD showed it needs $Ω( \sqrt{T} )$ communications for $T$ local gradient steps in order for the error to scale proportionately to $1/(NT)$, this has been successively improved in a string of papers, with the state of the art requiring $Ω\left( N \left( \mbox{ poly} (\log T) ight) ight)$ communications. In this paper, we suggest a Local SGD scheme that communicates less overall by communicating less frequently as the number of iterations grows. Our analysis shows that this can achieve an error that scales as $1/(NT)$ with a number of communications that is completely independent of $T$. In particular, we show that $Ω(N)$ communications are sufficient. Empirical evidence suggests this bound is close to tight as we further show that $\sqrt{N}$ or $N^{3/4}$ communications fail to achieve linear speed-up in simulations. Moreover, we show that under mild assumptions, the main of which is twice differentiability on any neighborhood of the optimal solution, one-shot averaging which only uses a single round of communication can also achieve the optimal convergence rate asymptotically.

研究动机与目标

• 在保持与工作节点数量线性加速的前提下，降低并行随机梯度下降（SGD）中的通信开销。
• 通过分析与迭代次数$T$无关的通信调度，解决局部SGD中通信频率与残差误差之间的权衡问题。
• 建立理论边界，表明$\Omega(N)$次通信足以在局部SGD中实现最优收敛速率$1/(NT)$。
• 探究单次平均（仅在最后通信一次）是否能实现与同步SGD相同的渐近收敛速率。

提出的方法

• 提出一种局部SGD方案，随着迭代次数增加而降低通信频率，采用递减学习率$\eta_t = 2t/(t+1)^2$以平衡收敛性与误差累积。
• 利用强凸参数$\mu$与梯度噪声方差$\sigma^2$，通过包含递归不等式的期望误差$\mathbb{E}[\|\bar{\mathbf{x}}^t - \mathbf{x}^*\|]$进行分析，误差项通过乘积不等式进行有界控制。
• 推导出三部分误差的边界：初始误差衰减（$S_1$）、方差引起的误差（$S_2$）与残差误差（$S_3$），并证明$S_3 = o(1/k)$且$S_2 \sim 1/k$。
• 使用不等式$\prod_{l=t_1}^{t_2-1}(1 - \eta_l\mu) \leq (t_1/t_2)^2$控制误差随时间的衰减，从而在最终误差率中引入$k$的依赖性。
• 在单次平均分析中采用迭代值的加权平均，证明在二阶可微性及Polyak-Łojasiewicz条件假设下可实现渐近线性加速。
• 采用两阶段分析：初始瞬态阶段（$t < t_0$）与渐近阶段（$t \geq t_0$），其中$t_0$的选择使得$\eta_t \approx 2t/(t+1)^2$能确保收敛。

实验结果

研究问题

• RQ1 局部SGD能否在与$T$无关的通信次数下实现最优的$1/(NT)$收敛速率？
• RQ2 实现与工作节点数$N$线性加速所需的最少通信轮数是多少？
• RQ3 单次平均（仅在最后通信一次）能否实现与同步SGD相同的渐近收敛速率？
• RQ4 局部更新产生的残差误差如何随通信频率与迭代次数变化？
• RQ5 在何种条件下，单次平均能实现与工作节点数$N$的线性加速？

主要发现

• 所提出的局部SGD方案实现误差率为$\mathbb{E}[\|\bar{\mathbf{x}}^k - \mathbf{x}^*\|^2] = \frac{4\sigma^2}{3N\mu^2 k} + o(1/k)$，与$T = k$次迭代下的最优$1/(NT)$标度一致。
• 仅需$\Omega(N)$次通信即可实现该最优收敛速率，且与总迭代次数$T$无关，显著优于以往依赖$\text{poly}(\log T)$或$\sqrt{T}$的通信界。
• 实验结果表明，$\sqrt{N}$或$N^{3/4}$次通信无法实现线性加速，暗示$\Omega(N)$接近紧致界。
• 在温和假设下（包括最优解邻域内的二阶可微性及Polyak-Łojasiewicz条件），单次平均可实现渐近线性加速。
• 残差误差项$\nu^t$被证明以$o(1/t^2)$的速率衰减，使得$S_3$项可忽略为$o(1/k)$，这对最终收敛速率至关重要。
• 分析确认，学习率$\eta_t = 2t/(t+1)^2$能确保误差项$S_1$充分衰减，并控制方差项$S_2$以实现$1/k$的收敛速率。