[Paper Review] MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms

Shaohuai Shi, Xiaowen Chu+1 more|arXiv (Cornell University)|Nov 27, 2018

Stochastic Gradient Optimization Techniques25 references4 citations

TL;DR

This paper proposes MG-WFBP, an optimized communication scheduling algorithm for distributed synchronous SGD that merges small gradient messages across layers to reduce network startup overhead. By formulating the problem as an optimization to minimize training iteration time, MG-WFBP achieves up to 1.75× speedup over WFBP and 1.45× over SyncEASGD on a 64-node GPU cluster, significantly improving scaling efficiency in high-communication-to-computation ratio scenarios.

ABSTRACT

Distributed synchronous stochastic gradient descent has been widely used to train deep neural networks on computer clusters. With the increase of computational power, network communications have become one limiting factor on system scalability. In this paper, we observe that many deep neural networks have a large number of layers with only a small amount of data to be communicated. Based on the fact that merging some short communication tasks into a single one may reduce the overall communication time, we formulate an optimization problem to minimize the training iteration time. We develop an optimal solution named merged-gradient WFBP (MG-WFBP) and implement it in our open-source deep learning platform B-Caffe. Our experimental results on an 8-node GPU cluster with 10GbE interconnect and trace-based simulation results on a 64-node cluster both show that the MG-WFBP algorithm can achieve much better scaling efficiency than existing methods WFBP and SyncEASGD.

Motivation & Objective

• Address the performance bottleneck in distributed deep learning caused by high communication-to-computation ratios, especially on high-bandwidth clusters with fast GPUs.
• Identify that existing methods like WFBP and SyncEASGD are suboptimal due to inefficient handling of small, frequent gradient transfers with high startup costs.
• Formulate the communication scheduling problem as an optimization to minimize total training iteration time, considering both computation-computation overlap and communication efficiency.
• Develop an optimal, globally efficient communication strategy that balances overlapping communication with computation and minimizes total communication time through intelligent message merging.
• Demonstrate the scalability and performance gains of MG-WFBP in real and simulated large-scale GPU clusters compared to state-of-the-art communication methods.

Proposed method

• Formulate the communication scheduling problem as an optimization to minimize training iteration time, considering layer-wise computation and communication costs.
• Propose MG-WFBP (Merged-Gradient Wait-Free Backpropagation), an algorithm that groups small gradient communications across layers into larger, more efficient messages to reduce network startup overhead.
• Design a dynamic programming-based solution with O(L²) time complexity to compute the optimal merging schedule across L layers, executed once before training.
• Integrate MG-WFBP into the open-source B-Caffe deep learning platform to enable end-to-end distributed training with optimized communication.
• Leverage wait-free backpropagation principles to allow communication to begin as soon as gradients are available, while ensuring merged messages do not delay computation.
• Use trace-based simulation on a 64-node cluster to evaluate scalability beyond physical hardware limits, validating performance under high-node-count conditions.

Experimental results

Research questions

• RQ1 Can merging small, frequent gradient communications across layers reduce overall training iteration time in distributed synchronous SGD?
• RQ2 Is the wait-free backpropagation (WFBP) strategy suboptimal when network startup costs dominate due to high communication-to-computation ratios?
• RQ3 Can a single-layer communication approach like SyncEASGD outperform WFBP in large-scale clusters with high startup overhead?
• RQ4 What is the optimal trade-off between overlapping communication with computation and minimizing total communication time through message merging?
• RQ5 Does a globally optimal communication scheduling strategy exist that outperforms both layer-wise and single-layer communication methods in terms of scaling efficiency?

Key findings

• MG-WFBP achieves up to 1.36× speedup over WFBP and 1.2× speedup over SyncEASGD on an 8-node GPU cluster with 10GbE interconnect.
• In trace-based simulation on a 64-node cluster, MG-WFBP achieves more than 1.7× speedup over WFBP and 1.3× speedup over SyncEASGD.
• On ResNet-50 training, MG-WFBP achieves approximately 1.75× and 1.45× speedups compared to WFBP and SyncEASGD, respectively, in the 64-node simulated environment.
• MG-WFBP maintains superior scaling efficiency—over 70% on the 64-node cluster—while WFBP and SyncEASGD degrade to around 55% due to unhidden startup overhead.
• The performance crossover between WFBP and SyncEASGD observed in simulations confirms that WFBP benefits from overlap in smaller clusters, but SyncEASGD and MG-WFBP outperform it in large-scale settings due to reduced startup costs.
• MG-WFBP's optimal message merging strategy enables better utilization of network bandwidth and reduces idle time, making it the most scalable solution under high communication-to-computation ratios.