[论文解读] Decentralized SGD with Asynchronous, Local and Quantized Updates

Giorgi Nadiradze, Amirmojtaba Sabour+4 more|arXiv (Cornell University)|May 4, 2021

Stochastic Gradient Optimization Techniques参考文献 40被引用 5

一句话总结

该论文提出了一种去中心化的SGD算法，统一了异步、本地和量化更新，以在最小协调的前提下扩展大规模神经网络训练。该算法在标准假设下证明了收敛性，并展示了实际可扩展性，其准确度达到或超过先前方法。

ABSTRACT

The ability to scale distributed optimization to large node counts has been one of the main enablers of recent progress in machine learning. To this end, several techniques have been explored, such as asynchronous, quantized and decentralized communication--which significantly reduce the impact of communication and synchronization, as well as the ability for nodes to perform several local model updates before communicating--which reduces the frequency of communication. In this paper, we show that these techniques, which have so far largely been considered independently, can be jointly leveraged to minimize distribution cost for training neural network models via stochastic gradient descent (SGD). We consider a setting with minimal coordination: we have a large number of nodes on a communication graph, each with a local subset of data, performing independent SGD updates onto their local models. After some number of local updates, each node chooses an interaction partner uniformly at random from its neighbors, and averages a (possibly quantized) version of its local model with the neighbor's model. Our first contribution is in proving that, even under such a relaxed setting, SGD can still be guaranteed to converge under standard assumptions. The proof is based on a new connection with parallel load-balancing processes, and improves existing techniques by handling decentralization, asynchrony, quantization, and local updates, into a single framework, and bounding their impact. On the practical side, we implement variants of our algorithm and deploy them onto distributed environments, and show that they can successfully converge and scale for large-scale neural network training tasks, matching or even slightly improving the accuracy of previous methods.

研究动机与目标

• 解决在最小协调的前提下将分布式深度学习训练扩展到大规模节点数量的挑战。
• 将异步更新、本地模型优化和通信量化整合到单一去中心化框架中。
• 在不牺牲收敛性的情况下，减少大规模神经网络训练中的通信和同步开销。
• 在放宽协调、异步性、量化和本地更新的条件下，为去中心化SGD建立理论收敛保证。
• 在真实分布式环境中展示所提方法在大规模模型训练中的实际有效性。

提出的方法

• 各节点在其本地数据子集上独立执行本地SGD更新，之后进行通信。
• 在完成固定次数的本地更新后，每个节点从其通信图中随机选择一个邻居，交换模型参数。
• 交换的模型参数经过平均处理，且平均操作在模型的量化版本上执行，以降低通信成本。
• 该算法以完全去中心化的方式运行，无需中央参数服务器或全局同步机制。
• 理论分析将更新过程与并行负载均衡动态联系起来，以限制去中心化、异步性、量化和本地更新的影响。
• 该框架支持任意通信图，并可处理确定性和随机量化方案。

实验结果

研究问题

• RQ1 在标准假设下，具有异步、本地和量化更新的去中心化SGD是否仍能保证收敛？
• RQ2 去中心化、异步性、量化和本地更新如何共同影响SGD的收敛行为？
• RQ3 将这些技术整合到单一统一框架中，对分布式优化的理论影响是什么？
• RQ4 所提方法在实践中是否能有效扩展至大规模神经网络训练，且无需集中协调？
• RQ5 与现有集中式或部分集中式方法相比，该方法的准确度和收敛速度如何？

主要发现

• 所提出的具有异步、本地和量化更新的去中心化SGD算法，在非凸优化的标准假设下被证明是收敛的。
• 理论分析建立了更新动态与并行负载均衡过程之间的新联系，从而实现了对收敛误差的更紧约束。
• 该方法即使在任意通信图、随机邻居选择和量化通信下仍能实现收敛。
• 实验评估表明，该算法在分布式环境中具有良好的可扩展性，并在准确度上达到或略微优于先前方法。
• 该框架显著降低了通信成本和同步开销，同时保持了训练效率和模型性能。