[Paper Review] Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback

Shuai Zheng, Ziyue Huang+1 more|arXiv (Cornell University)|May 27, 2019

Advanced Neural Network Applications28 references41 citations

TL;DR

This paper introduces dist-EF-SGD and blockwise variants to perform two-way gradient compression with error feedback and momentum, achieving about 32x communication reduction while preserving convergence rates for nonconvex problems, matching full-precision distributed SGD/SGDM in practice.

ABSTRACT

Communication overhead is a major bottleneck hampering the scalability of distributed machine learning systems. Recently, there has been a surge of interest in using gradient compression to improve the communication efficiency of distributed neural network training. Using 1-bit quantization, signSGD with majority vote achieves a 32x reduction on communication cost. However, its convergence is based on unrealistic assumptions and can diverge in practice. In this paper, we propose a general distributed compressed SGD with Nesterov's momentum. We consider two-way compression, which compresses the gradients both to and from workers. Convergence analysis on nonconvex problems for general gradient compressors is provided. By partitioning the gradient into blocks, a blockwise compressor is introduced such that each gradient block is compressed and transmitted in 1-bit format with a scaling factor, leading to a nearly 32x reduction on communication. Experimental results show that the proposed method converges as fast as full-precision distributed momentum SGD and achieves the same testing accuracy. In particular, on distributed ResNet training with 7 workers on the ImageNet, the proposed algorithm achieves the same testing accuracy as momentum SGD using full-precision gradients, but with $46\%$ less wall clock time.

Motivation & Objective

• Motivate and address communication bottlenecks in distributed SGD/SGDM for large-scale deep learning.
• Develop two-way gradient compression with error-feedback in a parameter-server setup.
• Propose blockwise gradient compression to improve compression quality while preserving convergence.
• Establish theoretical convergence guarantees for dist-EF-SGD and dist-EF-SGDM on nonconvex objectives.
• Validate the approach with experiments on ResNet/ImageNet and CIFAR-100 showing comparable accuracy to full-precision training.

Proposed method

• Extend EF-SGD to distributed settings with gradient compression and error feedback on both workers and server.
• Introduce dist-EF-SGD and its blockwise variant dist-EF-blockSGD, with two-way compression and error-correction; include rescaling of local/global error terms for changing step sizes.
• Provide convergence analysis under standard assumptions; prove O(1/√(MT)) rates matching fully-precise distributed SGD for nonconvex problems.
• Introduce blockwise compressor C_B that partitions gradients into blocks and compresses each block by a block-specific scaling factor to maintain a higher δ, enabling ~32x communication reduction.
• Extend to Nesterov momentum with dist-EF-blockSGDM; derive convergence results and discuss trade-offs with momentum on compression noise.
• Optionally include a momentum-enabled variant that follows a similar two-way compression framework.

Experimental results

Research questions

• RQ1 Can two-way gradient compression with error-feedback in a distributed parameter-server setting achieve convergence guarantees on nonconvex objectives?
• RQ2 How does blockwise compression affect compression quality and convergence compared to one-shot sign-based methods?
• RQ3 What are the convergence rates of dist-EF-SGD and dist-EF-SGDM with constant/decreasing/increasing stepsizes in nonconvex learning?
• RQ4 How does the proposed blockwise compressor improve δ and thereby convergence compared to standard 1-bit sign-based compression?
• RQ5 Do the proposed methods maintain accuracy while significantly reducing communication in large-scale experiments (e.g., ImageNet with ResNet)?

Key findings

• dist-EF-SGD achieves O(1/√(MT)) convergence rate under standard assumptions, matching distributed SGD with full-precision gradients.
• dist-EF-SGDM with Nesterov momentum attains O(1/√(MT)) convergence rate as well.
• Blockwise compressor C_B is a φ(v)-approximate compressor with φ(v) ≥ min_b 1/d_b, enabling nearly 32x communication reduction.
• Empirical results show testing accuracy comparable to full-precision momentum SGD, with significant wall-clock time savings on ImageNet/ResNet-50 (46% faster).
• SignSGD-based methods exhibit poorer accuracy in the reported experiments, highlighting the robustness of EF-based and block-EF methods.