[Paper Review] AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training

Chia Yu Chen, Jungwook Choi+4 more|arXiv (Cornell University)|Dec 7, 2017

Advanced Neural Network Applications74 citations

TL;DR

AdaComp introduces adaptive, localized residual gradient compression that self-tunes its rate across mini-batches and layers, achieving ~200x end-to-end compression for FC/LSTM and ~40x for conv layers with negligible accuracy loss.

ABSTRACT

Highly distributed training of Deep Neural Networks (DNNs) on future compute platforms (offering 100 of TeraOps/s of computational capacity) is expected to be severely communication constrained. To overcome this limitation, new gradient compression techniques are needed that are computationally friendly, applicable to a wide variety of layers seen in Deep Neural Networks and adaptable to variations in network architectures as well as their hyper-parameters. In this paper we introduce a novel technique - the Adaptive Residual Gradient Compression (AdaComp) scheme. AdaComp is based on localized selection of gradient residues and automatically tunes the compression rate depending on local activity. We show excellent results on a wide spectrum of state of the art Deep Learning models in multiple domains (vision, speech, language), datasets (MNIST, CIFAR10, ImageNet, BN50, Shakespeare), optimizers (SGD with momentum, Adam) and network parameters (number of learners, minibatch-size etc.). Exploiting both sparsity and quantization, we demonstrate end-to-end compression rates of ~200X for fully-connected and recurrent layers, and ~40X for convolutional layers, without any noticeable degradation in model accuracies.

Motivation & Objective

• Address the communication bottleneck in data-parallel distributed DNN training.
• Develop a universally applicable, computationally friendly gradient compression scheme for diverse layer types.
• Automatically adapt compression rate across mini-batches, layers, and hyper-parameters without extra tuning.
• Leverage sparsity and quantization to achieve high end-to-end compression while preserving convergence.
• Demonstrate robustness across datasets, architectures, and optimizers.

Proposed method

• Divide layer residual vectors into fixed-size bins and compute per-bin maxima to identify important gradients.
• Within each bin, select not only the local maximum but also nearby gradients using a self-adjusting threshold based on a scale factor (empirically 2x) applied to the residuals.
• Quantize the selected gradients to increase compression while sending a residual to maintain convergence.
• Apply AdaComp independently to each layer; learners exchange compressed gradients and then average across learners to update weights.
• Use a single layer-wide scale value (average of absolute values of g_max) for quantization representation (sign + scale).
• The method requires only one new hyper-parameter (L_T) to control bin size and achieves O(N) computation without global sorting.

Experimental results

Research questions

• RQ1 How can gradient communication be compressed in a way that is robust to layer type (FC, conv, recurrent) and network architecture?
• RQ2 Can a locally adaptive, threshold-based gradient selection scheme preserve convergence across diverse datasets and optimizers?
• RQ3 What compression rates are achievable end-to-end for different layer types without degrading accuracy?
• RQ4 How do mini-batch size and number of learners affect the attainable compression with AdaComp?
• RQ5 Is the approach scalable to large distributed systems while remaining accelerator-friendly?

Key findings

• AdaComp achieves ~200x end-to-end compression for fully-connected and recurrent layers and ~40x for convolutional layers.
• Compression preserves model accuracy across CNNs, DNNs, and LSTMs on MNIST, CIFAR10, ImageNet, BN50, and Shakespeare datasets.
• The method maintains convergence with SGD and Adam optimizers across varying mini-batch sizes and numbers of learners.
• AdaComp is robust to ultra-high compression rates due to its self-adjusting threshold, unlike prior schemes which diverge under heavy compression.
• Increasing mini-batch size lowers achievable compression, while more learners can increase compression for AdaComp with maintained accuracy.