[Paper Review] 99% of Distributed Optimization is a Waste of Time: The Issue and How to Fix it

Konstantin Mishchenko, Filip Hanzely+1 more|arXiv (Cornell University)|Jan 27, 2019

Stochastic Gradient Optimization Techniques41 references5 citations

TL;DR

This paper identifies that 99% of communication in standard distributed optimization is redundant due to transmitting full gradients across all workers, leading to inefficiency. It proposes a block-sparse gradient update method where each worker transmits only a fraction of gradient blocks, preserving convergence speed while reducing communication by up to 99%, validated by theory and experiments on synthetic and real datasets.

ABSTRACT

Many popular distributed optimization methods for training machine learning models fit the following template: a local gradient estimate is computed independently by each worker, then communicated to a master, which subsequently performs averaging. The average is broadcast back to the workers, which use it to perform a gradient-type step to update the local version of the model. It is also well known that many such methods, including SGD, SAGA, and accelerated SGD for over-parameterized models, do not scale well with the number of parallel workers. In this paper we observe that the above template is fundamentally inefficient in that too much data is unnecessarily communicated by the workers, which slows down the overall system. We propose a fix based on a new update-sparsification method we develop in this work, which we suggest be used on top of existing methods. Namely, we develop a new variant of parallel block coordinate descent based on independent sparsification of the local gradient estimates before communication. We demonstrate that with only $m/n$ blocks sent by each of $n$ workers, where $m$ is the total number of parameter blocks, the theoretical iteration complexity of the underlying distributed methods is essentially unaffected. As an illustration, this means that when $n=100$ parallel workers are used, the communication of $99\%$ blocks is redundant, and hence a waste of time. Our theoretical claims are supported through extensive numerical experiments which demonstrate an almost perfect match with our theory on a number of synthetic and real datasets.

Motivation & Objective

• To identify why popular distributed optimization methods like SGD, SAGA, and accelerated SGD do not scale well with increasing parallel workers.
• To demonstrate that the standard communication pattern—transmitting full gradients from all workers—is fundamentally inefficient and redundant.
• To develop a new update-sparsification technique that reduces communication overhead while preserving theoretical convergence rates.
• To show that only m/n blocks need to be communicated per worker (where m is total blocks, n is workers), making 99% of communication redundant when n=100.
• To validate the theoretical claims with extensive numerical experiments on synthetic and real-world datasets.

Proposed method

• Proposes a new variant of parallel block coordinate descent that applies independent sparsification to local gradient estimates before communication.
• Each worker computes a local gradient estimate and then selects only m/n blocks (where m is total parameter blocks) for transmission to the master.
• The master aggregates the sparsified blocks and broadcasts the average back to all workers for model updates.
• Uses a theoretical framework based on variance reduction and block-wise expectation to show that iteration complexity remains essentially unchanged despite reduced communication.
• Applies the method on top of existing algorithms like SAGA and SGD, enabling communication reduction without altering the core optimization logic.
• Employs a tower property and conditional expectation analysis to bound the expected error and prove convergence under standard assumptions (convexity, smoothness, bounded variance).

Experimental results

Research questions

• RQ1 Why do standard distributed optimization methods fail to scale linearly with the number of parallel workers?
• RQ2 To what extent is the full gradient communication in distributed methods redundant?
• RQ3 Can communication be reduced by a factor of 99% without degrading convergence speed?
• RQ4 What sparsification strategy preserves convergence guarantees in distributed block coordinate descent?
• RQ5 How does the proposed method compare to existing methods in terms of communication efficiency and iteration complexity?

Key findings

• When n=100 workers are used, transmitting 99% of gradient blocks is redundant, as only m/n blocks per worker are needed to maintain convergence speed.
• Theoretical analysis shows that the iteration complexity of the underlying distributed methods is essentially unaffected by reducing communication to m/n blocks per worker.
• Numerical experiments on synthetic and real datasets show an almost perfect match between theory and practice, confirming the effectiveness of the method.
• The proposed sparsification method enables a 99% reduction in communication overhead while maintaining the same convergence rate as full-gradient methods.
• The method is compatible with existing algorithms like SAGA, SGD, and accelerated SGD, allowing plug-in integration to improve scalability.
• Theoretical bounds show that the expected error decays at a rate comparable to full-communication methods, with only a small additive term due to variance from sparsification.