[Paper Review] Optimal Gradient Quantization Condition for Communication-Efficient Distributed Training

An Xu, Zhouyuan Huo+1 more|arXiv (Cornell University)|Feb 25, 2020

Sparse and Compressive Sensing Techniques37 references4 citations

TL;DR

This paper proposes BinGrad and ORQ, novel gradient quantization schemes that dynamically determine optimal quantization levels for binary and multi-level compression based on the gradient distribution, minimizing quantization variance. Experiments on CIFAR and ImageNet show superior model accuracy with lower communication costs, especially under aggressive quantization and distributed training settings.

ABSTRACT

The communication of gradients is costly for training deep neural networks with multiple devices in computer vision applications. In particular, the growing size of deep learning models leads to higher communication overheads that defy the ideal linear training speedup regarding the number of devices. Gradient quantization is one of the common methods to reduce communication costs. However, it can lead to quantization error in the training and result in model performance degradation. In this work, we deduce the optimal condition of both the binary and multi-level gradient quantization for extbf{ANY} gradient distribution. Based on the optimal condition, we develop two novel quantization schemes: biased BinGrad and unbiased ORQ for binary and multi-level gradient quantization respectively, which dynamically determine the optimal quantization levels. Extensive experimental results on CIFAR and ImageNet datasets with several popular convolutional neural networks show the superiority of our proposed methods.

Motivation & Objective

• To derive the optimal quantization condition for gradient compression without assuming a specific gradient distribution.
• To design practical quantization schemes that dynamically adapt quantization levels based on real-time gradient statistics.
• To minimize quantization-induced variance and bias in distributed deep learning training.
• To achieve better model performance with lower communication overhead, especially under aggressive compression.
• To validate the method in both single-machine and multi-worker distributed settings.

Proposed method

• Derives the optimal quantization condition for 2-level and multi-level gradient quantization using the full gradient distribution, without distributional assumptions.
• Proposes BinGrad-b and BinGrad-pb for binary quantization, differing in bias-variance trade-off: BinGrad-b minimizes variance, BinGrad-pb reduces bias with wider range.
• Introduces ORQ (Optimized Random Quantization) for multi-level quantization, using a greedy algorithm to select quantization levels that minimize expected quantization variance.
• Employs random rounding with learned thresholds to preserve gradient expectation and reduce error.
• Applies gradient clipping in distributed settings to further improve performance, especially for ORQ with fewer levels.
• Uses bucketed gradient compression with fixed-size chunks (e.g., d=512) to reduce per-bucket communication cost.

Experimental results

Research questions

• RQ1 What is the optimal condition for 2-level gradient quantization under any gradient distribution?
• RQ2 How can multi-level gradient quantization be optimized to minimize variance without assuming Gaussian or Laplace distributions?
• RQ3 Can dynamic, distribution-aware quantization levels outperform fixed-level schemes like QSGD and TernGrad?
• RQ4 How does the proposed method perform under aggressive compression and in distributed training with limited bandwidth?
• RQ5 Does the proposed scheme maintain model accuracy while reducing communication overhead compared to baseline methods?

Key findings

• ORQ achieves 0.36% to 2.85% higher top-1 accuracy than baseline methods on CIFAR-10/100 under no clipping and large batch settings.
• With bucket size increasing from 128 to 32768, ORQ degrades only 4.58% in test accuracy, compared to 5.23% for the baseline, showing greater robustness.
• On ImageNet, ORQ-5 and ORQ-9 achieve test accuracy within 0.8% of FP gradients, with ORQ-3 matching QSGD-5 and QSGD-9 performance.
• Reducing compression ratio from 20.2 to 10.1 improves top-1 accuracy by 1.47% for ORQ, compared to 0.95% for the baseline, indicating better efficiency.
• BinGrad-b achieves lower quantization error than BinGrad-pb, and both outperform 3-level quantization in test accuracy when no clipping is used.
• The proposed methods maintain high performance even with aggressive quantization, demonstrating resilience to communication constraints in distributed training.