[Paper Review] TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning

Wei Wen, Cong Xu+5 more|arXiv (Cornell University)|May 22, 2017

Robotics and Automated Systems419 citations

TL;DR

TernGrad quantizes gradients to ternary values (-1, 0, 1) to cut communication in distributed data-parallel training, with convergence guarantees and layer-wise techniques to improve performance; experiments show minimal to no accuracy loss and notable speedups.

ABSTRACT

High network communication cost for synchronizing gradients and parameters is the well-known bottleneck of distributed training. In this work, we propose TernGrad that uses ternary gradients to accelerate distributed deep learning in data parallelism. Our approach requires only three numerical levels {-1,0,1}, which can aggressively reduce the communication time. We mathematically prove the convergence of TernGrad under the assumption of a bound on gradients. Guided by the bound, we propose layer-wise ternarizing and gradient clipping to improve its convergence. Our experiments show that applying TernGrad on AlexNet does not incur any accuracy loss and can even improve accuracy. The accuracy loss of GoogLeNet induced by TernGrad is less than 2% on average. Finally, a performance model is proposed to study the scalability of TernGrad. Experiments show significant speed gains for various deep neural networks. Our source code is available.

Motivation & Objective

• Motivate and address the communication bottleneck in distributed SGD for data-parallel deep learning.
• Propose a ternary gradient quantization method to replace full-precision gradient synchronization.
• Develop theoretically grounded bounds and practical techniques to ensure convergence and stability.
• Demonstrate empirical accuracy preservation (or gain) and measure scalability and speedups on standard DNNs.

Proposed method

• Quantize gradients to ternary values using a random Bernoulli-based masking guided by gradient magnitudes.
• Apply a shared scalar s_t to adapt the ternary values and achieve unbiased gradient estimates.
• Use parameter localization to replace server-side parameter synchronization with pulling quantized gradients.
• Introduce layer-wise ternarizing and gradient clipping to tighten convergence bounds and reduce gradient range.
• Provide a convergence analysis showing almost sure convergence under standard online gradient conditions and a gradient bound.

Experimental results

Research questions

• RQ1 Can ternary gradient quantization ensure convergence of distributed SGD?
• RQ2 How do layer-wise ternarizing and gradient clipping affect convergence and practical performance?
• RQ3 What accuracy and speedups are achievable on standard CNN architectures when using TernGrad?
• RQ4 How does TernGrad scale with the number of workers and network bandwidth?

Key findings

• TernGrad converges to the minimum almost surely under the proposed assumptions and with the ternary gradient estimator.
• Layer-wise ternarizing and gradient clipping tighten the convergence bound and improve stability in practice.
• AlexNet shows no accuracy loss and can even improve accuracy under TernGrad; GoogLeNet incurs less than ~2% average top-1 loss.
• Empirical results demonstrate significant training speedups due to reduced communication, especially for networks with high communication-to-computation ratios.
• A performance model indicates notable throughput gains on multi-GPU clusters and across varying bandwidths.