[Paper Review] An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems

Ahmed M. Abdelmoniem, Ahmed Elzanaty+2 more|arXiv (Cornell University)|Jan 26, 2021

Advanced Neural Network Applications71 references31 citations

TL;DR

SIDCo introduces a multi-stage, threshold-based gradient sparsification method using sparsity-inducing distributions to accurately estimate compression thresholds with low overhead, accelerating distributed training.

ABSTRACT

The recent many-fold increase in the size of deep neural networks makes efficient distributed training challenging. Many proposals exploit the compressibility of the gradients and propose lossy compression techniques to speed up the communication stage of distributed training. Nevertheless, compression comes at the cost of reduced model quality and extra computation overhead. In this work, we design an efficient compressor with minimal overhead. Noting the sparsity of the gradients, we propose to model the gradients as random variables distributed according to some sparsity-inducing distributions (SIDs). We empirically validate our assumption by studying the statistical characteristics of the evolution of gradient vectors over the training process. We then propose Sparsity-Inducing Distribution-based Compression (SIDCo), a threshold-based sparsification scheme that enjoys similar threshold estimation quality to deep gradient compression (DGC) while being faster by imposing lower compression overhead. Our extensive evaluation of popular machine learning benchmarks involving both recurrent neural network (RNN) and convolution neural network (CNN) models shows that SIDCo speeds up training by up to 41:7%, 7:6%, and 1:9% compared to the no-compression baseline, Topk, and DGC compressors, respectively.

Motivation & Objective

• Motivate and address the communication bottleneck in distributed DNN training.
• Model gradients as sparsity-inducing distributions to enable efficient compression.
• Develop a multi-stage, threshold-based compression scheme with low overhead.
• Provide closed-form threshold estimators for SIDCo to achieve target compression.
• Demonstrate speedups and training efficiency gains across RNN and CNN benchmarks.

Proposed method

• Model gradients as random variables drawn from sparsity-inducing distributions (SID): double exponential, double gamma, and double generalized Pareto.
• Derive a threshold to achieve a target compression ratio using inverse CDFs of the absolute gradient distributions.
• Propose a single-stage thresholding scheme and a multi-stage estimator to improve far-tail threshold accuracy at aggressive sparsification.
• Use a multi-stage PoT (peak over threshold) fitting approach with Corollaries for exponential, gamma, and GP distributions to adapt thresholds.
• Provide an adaptive SIDCo algorithm that selects the number of stages M to bound estimation error.
• Analyze convergence, showing SIDCo’s rate matches SGD under bounded compression discrepancy.

Experimental results

Research questions

• RQ1 How can gradient compression be performed with minimal computational overhead while maintaining convergence performance?
• RQ2 Can gradient distributions be effectively modeled with sparsity-inducing distributions to enable accurate threshold estimation?
• RQ3 Does a multi-stage threshold estimator improve threshold accuracy for aggressive sparsification across architectures?
• RQ4 What are the convergence guarantees of SIDCo under threshold-based sparsification?
• RQ5 What practical speedups and quality trade-offs does SIDCo achieve on standard benchmarks?

Key findings

• SIDCo attains training speedups up to approximately 41.7x, 7.6x, and 1.9x over no-compression, Topk, and DGC compressors respectively.
• Single-stage thresholding with SID can achieve near-target compression with lower overhead than Topk or DGC in both GPU and CPU settings.
• Multi-stage threshold estimation improves tail accuracy for aggressive sparsification (δ very small).
• SIDCo’s convergence rate matches SGD under bounded discrepancy, implying no loss in asymptotic convergence behavior.
• Experimental results on RNN and CNN benchmarks demonstrate consistent performance gains and accurate threshold estimation across models.
• SIDCo’s complexity is linear in model size, enabling scalable GPU-parallel implementations.