[Paper Review] Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent

Shuheng Shen, Linli Xu+3 more|arXiv (Cornell University)|Jun 28, 2019

Stochastic Gradient Optimization Techniques24 references4 citations

TL;DR

This paper proposes CoCoD-SGD, a computation and communication decoupled stochastic gradient descent algorithm that runs computation and communication in parallel while reducing communication frequency, achieving up to 2-3× faster training than traditional synchronous SGD on 16 GPUs with ResNet18 and VGG16, and proving linear iteration speedup in both homogeneous and heterogeneous environments.

ABSTRACT

With the increase in the amount of data and the expansion of model scale, distributed parallel training becomes an important and successful technique to address the optimization challenges. Nevertheless, although distributed stochastic gradient descent (SGD) algorithms can achieve a linear iteration speedup, they are limited significantly in practice by the communication cost, making it difficult to achieve a linear time speedup. In this paper, we propose a computation and communication decoupled stochastic gradient descent (CoCoD-SGD) algorithm to run computation and communication in parallel to reduce the communication cost. We prove that CoCoD-SGD has a linear iteration speedup with respect to the total computation capability of the hardware resources. In addition, it has a lower communication complexity and better time speedup comparing with traditional distributed SGD algorithms. Experiments on deep neural network training demonstrate the significant improvements of CoCoD-SGD: when training ResNet18 and VGG16 with 16 Geforce GTX 1080Ti GPUs, CoCoD-SGD is up to 2-3$ imes$ faster than traditional synchronous SGD.

Motivation & Objective

• Address the communication bottleneck in distributed deep learning that limits linear time speedup despite high computational capacity.
• Overcome the inefficiency of traditional synchronous SGD, where computation and communication are tightly coupled and only partial hardware resources are utilized during communication.
• Develop a distributed optimization algorithm that enables parallel execution of computation and communication to improve hardware utilization.
• Achieve linear iteration speedup with respect to total computation capability in both homogeneous and heterogeneous hardware environments.
• Demonstrate superior time speedup and convergence efficiency in practice without sacrificing model accuracy.

Proposed method

• Decouple computation and communication by allowing workers to continue local gradient computation immediately after initiating communication, rather than waiting for communication completion.
• Introduce periodic communication instead of per-iteration communication, reducing overall communication complexity.
• Update local models using both the communicated global gradient and the difference between local and global models to maintain convergence stability.
• Theoretical analysis proves linear iteration speedup in homogeneous settings and with respect to total computation capability in heterogeneous settings.
• Support both homogeneous and heterogeneous environments by dynamically adjusting communication frequency and batch size based on worker speed.
• Implement a communication protocol that allows overlapping of computation and communication phases, minimizing idle time on workers.

Experimental results

Research questions

• RQ1 Can computation and communication be effectively decoupled in distributed SGD to improve hardware utilization and reduce communication overhead?
• RQ2 Does the proposed CoCoD-SGD algorithm achieve linear iteration speedup with respect to the number of workers in homogeneous environments?
• RQ3 Can CoCoD-SGD maintain linear iteration speedup with respect to total computation capability in heterogeneous environments?
• RQ4 What is the practical time speedup of CoCoD-SGD compared to existing distributed SGD variants like S-SGD, Pipe-SGD, and Local-SGD?
• RQ5 Does CoCoD-SGD preserve model accuracy while achieving faster convergence in both homogeneous and heterogeneous settings?

Key findings

• On 16 GPUs, CoCoD-SGD achieves up to 2-3× faster training than traditional synchronous SGD for ResNet18 and VGG16.
• CoCoD-SGD demonstrates superior time speedup, with 2.5× and 3× speedup against S-SGD for ResNet18 and VGG16 respectively in heterogeneous environments.
• The algorithm maintains or slightly improves test accuracy compared to S-SGD, with final test accuracy of 94.33% on CIFAR-10 with ResNet18 and 75.74% on CIFAR-100 with VGG16.
• In heterogeneous environments, CoCoD-SGD is robust to varying worker speeds and maintains fast convergence, outperforming S-SGD and Pipe-SGD even when enhanced with proportionally sampling.
• Theoretical analysis confirms that CoCoD-SGD achieves linear iteration speedup in both homogeneous and heterogeneous settings.
• Experiments show that CoCoD-SGD reduces communication complexity while enabling full utilization of hardware resources through overlapping computation and communication.