[Paper Review] Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization

Jiaxiang Wu, Weidong Huang+2 more|arXiv (Cornell University)|Jun 20, 2018

Distributed and Parallel Computing SystemsComputer Science111 citations

TL;DR

ECQ-SGD introduces accumulated error feedback to quantized gradients in data-parallel distributed optimization, achieving substantial gradient compression without performance loss and faster convergence than prior quantized SGD methods.

ABSTRACT

Large-scale distributed optimization is of great importance in various applications. For data-parallel based distributed learning, the inter-node gradient communication often becomes the performance bottleneck. In this paper, we propose the error compensated quantized stochastic gradient descent algorithm to improve the training efficiency. Local gradients are quantized to reduce the communication overhead, and accumulated quantization error is utilized to speed up the convergence. Furthermore, we present theoretical analysis on the convergence behaviour, and demonstrate its advantage over competitors. Extensive experiments indicate that our algorithm can compress gradients by a factor of up to two magnitudes without performance degradation.

Motivation & Objective

• Motivate the need for reducing inter-node communication in data-parallel distributed optimization.
• Propose ECQ-SGD that uses accumulated quantization error for compensation.
• Provide theoretical convergence analysis comparing ECQ-SGD to QSGD.
• Empirically validate ECQ-SGD on linear models and CNNs showing compression without loss of performance.

Proposed method

• Define an unbiased stochastic quantization function Q with a limited codebook and show how gradients are quantized.
• Introduce accumulated quantization error h_p^(t) per node and a compensation step using a coefficient alpha.
• Quantize compensated gradients tilde{g}_p^(t) = Q(g_p^(t) + alpha h_p^(t)).
• Update accumulated error h_p^(t+1) = beta h_p^(t) + (g_p^(t) - tilde{g}_p^(t)).
• Perform distributed update w^{(t+1)} = w^{(t)} - eta * (1/P) sum_p tilde{g}_p^(t).
• Provide a convergence analysis for quadratic objectives showing how ECQ-SGD suppresses quantization error contributions compared to QSGD.

Experimental results

Research questions

• RQ1 Can ECQ-SGD achieve comparable convergence to full-precision SGD while using quantized gradients?
• RQ2 Does the accumulated error compensation improve convergence speed and stability over prior quantized SGD methods (e.g., QSGD, 1Bit-SGD)?
• RQ3 How do the hyper-parameters alpha and beta affect convergence and stability in practice?
• RQ4 What is the impact of ECQ-SGD on communication cost and scalability in large GPU clusters?

Key findings

• ECQ-SGD can compress gradients by a factor of up to two magnitudes without performance degradation.
• The accumulated error compensation suppresses the quantization error’s contribution to the error bound, yielding faster and more stable convergence than baseline methods.
• For linear models, ECQ-SGD converges faster in loss value and closer to the optimal solution than QSGD across multiple synthetic datasets.
• Experiments on CIFAR-10 with ResNet-20 show ECQ-SGD achieves significant communication reduction with comparable accuracy.
• Performance modeling and scalability experiments demonstrate notable speed-ups on large GPU clusters; e.g., 512 GPUs yield substantial throughput gains over vanilla SGD.