[论文解读] Parallel Restarted SGD for Non-Convex Optimization with Faster Convergence and Less Communication.

Hao Yu, Sen Yang+1 more|arXiv (Cornell University)|Jul 17, 2018

Stochastic Gradient Optimization Techniques参考文献 15被引用 50

一句话总结

本文提出并行重启SGD，一种针对大规模非凸问题的通信高效优化方法，通过仅在周期性重启时交换模型平均值来减少工作节点间的通信。该方法在保持与经典并行小批量SGD相同收敛速率的同时，将通信开销降低了$O(T^{1/4})$倍，为深度学习中模型平均的实证成功提供了理论依据。

ABSTRACT

For large scale non-convex stochastic optimization, parallel mini-batch SGD using multiple workers ideally can achieve a linear speed-up with respect to the number of workers compared with SGD over a single worker. However, such linear scalability in practice is significantly limited by the growing demand for communication as more workers are involved. This is because the classical parallel mini-batch SGD requires gradient or model exchanges between workers (possibly through an intermediate server) at every iteration. In this paper, we study whether it is possible to maintain the linear speed-up property of parallel mini-batch SGD by using less frequent message passing between workers. We consider the parallel restarted SGD method where each worker periodically restarts its SGD by using the node average as a new initial point. Such a strategy invokes inter-node communication only when computing the node average to restart local SGD but otherwise is fully parallel with no communication overhead. We prove that the parallel restarted SGD method can maintain the same convergence rate as the classical parallel mini-batch SGD while reducing the communication overhead by a factor of $O(T^{1/4})$. The parallel restarted SGD strategy was previously used as a common practice, known as model averaging, for training deep neural networks. Earlier empirical works have observed that model averaging can achieve an almost linear speed-up if the averaging interval is carefully controlled. The results in this paper can serve as theoretical justifications for these empirical results on model averaging and provide practical guidelines for applying model averaging.

研究动机与目标

• 解决大规模非凸优化中并行小批量SGD的通信瓶颈问题。
• 探究降低通信频率是否能在不牺牲收敛速率的前提下维持线性加速。
• 为深度学习中模型平均的实证成功提供理论依据。
• 设计一种支持频繁本地更新、稀疏同步的方法，以提升可扩展性。

提出的方法

• 每个工作节点在迭代之间独立执行本地SGD更新，不进行通信。
• 在固定时间间隔，工作节点交换并平均其模型，以计算新的全局初始化点。
• 每个工作节点从平均后的模型重新开始其本地SGD，从而每几轮迭代实现一次有效同步。
• 该方法通过周期性重启来维持收敛性，而无需持续交换梯度。
• 理论分析表明，在标准假设下，其收敛速率与经典并行小批量SGD相当。
• 通信仅在模型平均步骤发生，与全频通信方法相比，总通信量降低了$O(T^{1/4})$。

实验结果

研究问题

• RQ1 在并行SGD中降低通信频率是否能保持与经典并行小批量SGD相同的收敛速率？
• RQ2 周期性模型平均对非凸优化中收敛性的理论影响是什么？
• RQ3 通信频率如何影响并行SGD的可扩展性和加速性能？
• RQ4 模型平均的观测到的实证成功能否从理论上得到解释？

主要发现

• 所提出的并行重启SGD在标准非凸优化假设下，实现了与经典并行小批量SGD相同的收敛速率。
• 与全通信并行SGD相比，通信开销降低了$O(T^{1/4})$倍，其中$T$为总迭代次数。
• 尽管通信频率降低，该方法仍能保持与工作节点数量成线性关系的加速性能。
• 理论分析证实，通过重启进行周期性模型平均已足够实现收敛，支持其在深度学习训练中的应用。