[Paper Review] Communication-Efficient Distributed Deep Learning: A Comprehensive Survey

Zhenheng Tang, Shaohuai Shi+3 more|arXiv (Cornell University)|Mar 10, 2020

Stochastic Gradient Optimization Techniques210 references89 citations

TL;DR

A comprehensive survey of communication-efficient data-parallel distributed deep learning, outlining a four-dimension taxonomy (synchronization, system architectures, compression, and communication-computation parallelism), benchmarks, and convergence analysis.

ABSTRACT

Distributed deep learning (DL) has become prevalent in recent years to reduce training time by leveraging multiple computing devices (e.g., GPUs/TPUs) due to larger models and datasets. However, system scalability is limited by communication becoming the performance bottleneck. Addressing this communication issue has become a prominent research topic. In this paper, we provide a comprehensive survey of the communication-efficient distributed training algorithms, focusing on both system-level and algorithmic-level optimizations. We first propose a taxonomy of data-parallel distributed training algorithms that incorporates four primary dimensions: communication synchronization, system architectures, compression techniques, and parallelism of communication and computing tasks. We then investigate state-of-the-art studies that address problems in these four dimensions. We also compare the convergence rates of different algorithms to understand their convergence speed. Additionally, we conduct extensive experiments to empirically compare the convergence performance of various mainstream distributed training algorithms. Based on our system-level communication cost analysis, theoretical and experimental convergence speed comparison, we provide readers with an understanding of which algorithms are more efficient under specific distributed environments. Our research also extrapolates potential directions for further optimizations.

Motivation & Objective

• Introduce a taxonomy for data-parallel distributed training focused on communication efficiency.
• Survey state-of-the-art synchronization, system architectures, compression techniques, and scheduling methods.
• Analyze convergence bounds and empirically compare mainstream algorithms.
• Provide a benchmark framework and practical guidance for choosing algorithms under different environments.

Proposed method

• Propose a four-dimensional taxonomy: communication synchronization, system architectures, compression techniques, and parallelism of communication and computing.
• Review BSP-SGD and variations, along with synchronous, stale-synchronous, asynchronous, and Local-SGD frameworks.
• Survey centralized (Parameter Server) and decentralized (All-Reduce, Gossip) architectures and their trade-offs.
• Organize and compare convergence analyses and provide empirical benchmarks using a FedML/MPI-based framework.
• Summarize auxiliary tricks and potential future directions for optimizing communication in distributed DL.

Experimental results

Research questions

• RQ1 What are the main dimensions that govern communication efficiency in data-parallel distributed deep learning?
• RQ2 How do different synchronization schemes, system architectures, compression techniques, and computation-communication parallelism affect convergence and scalability?
• RQ3 What are the theoretical convergence guarantees and practical performance trade-offs of representative algorithms across architectures?
• RQ4 How do empirical results compare across mainstream algorithms under unified experimental settings?
• RQ5 What future directions exist for further reducing communication bottlenecks in distributed DL?

Key findings

• A four-dimension taxonomy captures major axes of communication efficiency in data-parallel DL.
• Synchronization, architecture, compression, and recombined scheduling choices yield distinct trade-offs in convergence, congestion, and model consistency.
• Empirical experiments show BSP-SGD, ASP-SGD, Local-SGD, and FedAvg can reach similar accuracy under unified settings, but performance and stability vary with worker count and staleness.
• Asynchrony can improve wall-clock time yet may degrade convergence; local updates (Local-SGD) can reduce communication but require careful synchronization frequency to avoid accuracy loss.
• Centralized (PS) vs. decentralized (All-Reduce, Gossip) architectures present different bottlenecks, congestion profiles, and ease of scaling.