[Paper Review] Communication-Efficient Distributed Learning via Lazily Aggregated Quantized Gradients

Jun Sun, Tianyi Chen+2 more|arXiv (Cornell University)|Sep 17, 2019

Stochastic Gradient Optimization Techniques39 citations

TL;DR

LAQ introduces gradient innovation-based quantization and lazy aggregation to reduce both communication bits and rounds while maintaining linear convergence for strongly convex losses.

ABSTRACT

The present paper develops a novel aggregated gradient approach for distributed machine learning that adaptively compresses the gradient communication. The key idea is to first quantize the computed gradients, and then skip less informative quantized gradient communications by reusing outdated gradients. Quantizing and skipping result in `lazy' worker-server communications, which justifies the term Lazily Aggregated Quantized gradient that is henceforth abbreviated as LAQ. Our LAQ can provably attain the same linear convergence rate as the gradient descent in the strongly convex case, while effecting major savings in the communication overhead both in transmitted bits as well as in communication rounds. Empirically, experiments with real data corroborate a significant communication reduction compared to existing gradient- and stochastic gradient-based algorithms.

Motivation & Objective

• Motivate reducing communication bottlenecks in distributed learning with many workers.
• Develop a method that compresses gradients and reduces upload rounds without losing convergence guarantees.
• Achieve similar convergence to gradient descent while saving communication resources.
• Provide theoretical guarantees and empirical validation on real datasets.

Proposed method

• Quantize gradient innovations instead of full gradients to reduce bits per upload.
• Reuse outdated gradients at the server and upload only selected workers’ gradient innovations (lazy aggregation).
• Define a selection criterion to decide which workers upload in each iteration, based on gradient and parameter differences.
• Prove linear convergence of LAQ under strong convexity with quantization error.
• Analyze convergence via a Lyapunov function incorporating past iterates and quantization errors.

Experimental results

Research questions

• RQ1 Can gradient quantization combined with lazy aggregation achieve the same linear convergence as GD for strongly convex losses?
• RQ2 What is the impact of gradient innovation-based uploads on communication rounds and transmitted bits?
• RQ3 How to design a reliable criterion to decide which workers upload each iteration while preserving convergence?
• RQ4 What are the theoretical guarantees (rates) when quantization errors are present?
• RQ5 How does LAQ perform empirically on real datasets compared to GD, QGD, and LAG?

Key findings

• LAQ achieves linear convergence to the optimum under μ-strongly convex f(·) despite quantization and skipped uploads.
• LAQ reduces both communication rounds and total bits compared to GD and QGD, while preserving accuracy.
• Empirical results show LAQ requiring fewer communication rounds than GD and QGD and fewer bits than LAG for logistic regression and neural networks, with comparable accuracy.
• SLAQ (stochastic variant) further reduces communication rounds and bits, outperforming QSGD and SSGD in the tested scenarios.
• Theoretical analysis provides a Lyapunov-based convergence guarantee and bounds on quantization error growth.