[Paper Review] Elastic Gossip: Distributing Neural Network Training Using Gossip-like Protocols

Siddharth Pramod|arXiv (Cornell University)|Dec 6, 2018

Reinforcement Learning in Robotics23 references4 citations

TL;DR

This paper proposes Elastic Gossip, a novel gossip-like protocol for distributing neural network training that reduces communication overhead by enabling asynchronous, pairwise parameter averaging. It improves upon Gossiping SGD by incorporating an adaptive moving rate, achieving comparable performance to All-reduce SGD with significantly lower bandwidth usage, particularly on MNIST and CIFAR-10 benchmarks using MLPs and CNNs.

ABSTRACT

Distributing Neural Network training is of particular interest for several reasons including scaling using computing clusters, training at data sources such as IOT devices and edge servers, utilizing underutilized resources across heterogeneous environments, and so on. Most contemporary approaches primarily address scaling using computing clusters and require high network bandwidth and frequent communication. This thesis presents an overview of standard approaches to distribute training and proposes a novel technique involving pairwise-communication using Gossip-like protocols, called Elastic Gossip. This approach builds upon an existing technique known as Elastic Averaging SGD (EASGD), and is similar to another technique called Gossiping SGD which also uses Gossip-like protocols. Elastic Gossip is empirically evaluated against Gossiping SGD using the MNIST digit recognition and CIFAR-10 classification tasks, using commonly used Neural Network architectures spanning Multi-Layer Perceptrons (MLPs) and Convolutional Neural Networks (CNNs). It is found that Elastic Gossip, Gossiping SGD, and All-reduce SGD perform quite comparably, even though the latter entails a substantially higher communication cost. While Elastic Gossip performs better than Gossiping SGD in these experiments, it is possible that a more thorough search over hyper-parameter space, specific to a given application, may yield configurations of Gossiping SGD that work better than Elastic Gossip.

Motivation & Objective

• To address the high communication cost of distributed SGD in large-scale and heterogeneous environments.
• To improve upon existing gossip-based training methods like Gossiping SGD by introducing a dynamic moving rate mechanism.
• To evaluate whether a gossip-like protocol can achieve performance comparable to centralized All-reduce SGD while minimizing network bandwidth usage.
• To explore the feasibility of training deep neural networks efficiently on decentralized, edge-computing, or privacy-sensitive systems.

Proposed method

• Elastic Gossip uses a pairwise communication model where workers randomly select peers to exchange and average model parameters.
• It extends Elastic Averaging SGD (EASGD) by introducing a moving rate α that controls the degree of parameter averaging, enabling adaptive convergence.
• Each worker performs local SGD updates and occasionally synchronizes with a randomly chosen peer, using a Bernoulli-distributed communication probability p to avoid network congestion.
• The method incorporates Nesterov’s accelerated gradient (NAG) for faster convergence and uses a shared clock to ensure synchronization before communication.
• The algorithm dynamically updates parameters using a weighted average of local and peer parameters, with the weight controlled by α.
• Communication is decoupled from training steps via a probabilistic trigger (p), reducing network load while maintaining model consistency.

Experimental results

Research questions

• RQ1 Can a gossip-like protocol achieve training performance comparable to All-reduce SGD while significantly reducing communication overhead?
• RQ2 How does the moving rate α in Elastic Gossip affect convergence speed and final model accuracy?
• RQ3 Does Elastic Gossip outperform Gossiping SGD under the same hyperparameter settings and datasets?
• RQ4 How do communication probability p and number of workers influence the performance of gossip-based training?
• RQ5 To what extent can Elastic Gossip scale across heterogeneous or decentralized environments with limited bandwidth?

Key findings

• Elastic Gossip achieved test accuracies of 98.65% on MNIST and 87.4% on CIFAR-10, matching or slightly exceeding those of All-reduce SGD.
• Elastic Gossip outperformed Gossiping SGD across all tested configurations, particularly at lower communication probabilities.
• The optimal moving rate α was found to be 0.5, with performance degrading at both lower (0.1) and higher (0.9) values.
• Communication probability p had a significant impact: performance degraded when p dropped below 0.125 (effective period τ = 8), suggesting a trade-off between bandwidth and convergence.
• Despite using only pairwise communication, Elastic Gossip achieved performance comparable to All-reduce SGD, which requires full synchronization and high-bandwidth coordination.
• The results suggest that Elastic Gossip is a viable alternative to All-reduce SGD in bandwidth-constrained or decentralized training scenarios.