[Paper Review] Sparse-Push: Communication- & Energy-Efficient Decentralized Distributed Learning over Directed & Time-Varying Graphs with non-IID Datasets.

Sai Aparna Aketi, Amandeep Singh+1 more|arXiv (Cornell University)|Feb 10, 2021

Privacy-Preserving Technologies in Data14 references4 citations

TL;DR

This paper proposes Sparse-Push (SP) and Skew-Compensated Sparse Push (SCSP), communication- and energy-efficient decentralized training algorithms for directed, time-varying peer-to-peer graphs with non-IID data. The methods achieve up to 466× communication reduction with only 1% accuracy degradation on ResNet-20 and VGG11 over CIFAR-10, while SCSP recovers performance loss from compression on non-IID data.

ABSTRACT

Current deep learning (DL) systems rely on a centralized computing paradigm which limits the amount of available training data, increases system latency, and adds privacy and security constraints. On-device learning, enabled by decentralized and distributed training of DL models over peer-to-peer wirelessly connected edge devices, not only alleviate the above limitations but also enable next-gen applications that need DL models to continuously interact and learn from their environment. However, this necessitates the development of novel training algorithms that train DL models over time-varying and directed peer-to-peer graph structures while minimizing the amount of communication between the devices and also being resilient to non-IID data distributions. In this work we propose, Sparse-Push, a communication efficient decentralized distributed training algorithm that supports training over peer-to-peer, directed, and time-varying graph topologies. The proposed algorithm enables 466x reduction in communication with only 1% degradation in performance when training various DL models such as ResNet-20 and VGG11 over the CIFAR-10 dataset. Further, we demonstrate how communication compression can lead to significant performance degradation in-case of non-IID datasets, and propose Skew-Compensated Sparse Push algorithm that recovers this performance drop while maintaining similar levels of communication compression.

Motivation & Objective

• Address the limitations of centralized deep learning, including high communication costs, latency, and privacy risks in edge AI applications.
• Develop a decentralized training algorithm resilient to non-IID data distributions across edge devices.
• Minimize communication and energy usage in decentralized training over time-varying, directed peer-to-peer graphs.
• Overcome performance degradation caused by communication compression in non-IID settings.
• Enable on-device training on resource-constrained devices like smartphones and drones through reduced per-device memory and compute load.

Proposed method

• Propose Sparse-Push (SP), a communication-efficient decentralized optimization algorithm using top-K sparsification and push-sum-based averaging over directed, time-varying graphs.
• Introduce Skew-Compensated Sparse Push (SCSP), which dynamically adjusts sparsification weights to counteract performance drops from compression on non-IID data.
• Leverage a push-sum protocol to maintain consensus in directed graphs where connectivity is asymmetric and not necessarily doubly stochastic.
• Apply top-K sparsification to reduce communication by transmitting only the largest-magnitude gradients or model updates.
• Use a decentralized training framework that allows each node to control its own communication schedule and neighbors, without requiring global knowledge of the network topology.
• Design the algorithm to be resilient to time-varying connectivity and support asynchronous updates in future extensions.

Experimental results

Research questions

• RQ1 Can a decentralized training algorithm achieve high model accuracy while drastically reducing communication in directed, time-varying peer-to-peer graphs?
• RQ2 How does communication compression affect model performance under non-IID data distributions in decentralized learning?
• RQ3 Can performance degradation from compression on non-IID data be recovered without sacrificing communication efficiency?
• RQ4 To what extent can communication and energy costs be reduced in decentralized training for edge devices?
• RQ5 How does the proposed algorithm scale to larger models and datasets like CIFAR-100?

Key findings

• Sparse-Push achieves up to 466× reduction in communication with only 1% accuracy degradation when training ResNet-20 and VGG11 on CIFAR-10.
• On CIFAR-100 with ResNet-110, SCSP achieves a 438× communication reduction with a 1.62% accuracy gain over baseline compression.
• Communication compression causes significant performance drops on non-IID data, with accuracy dropping from 92.09% to 88.81% on VGG11 under 0.1% sparsification.
• SCSP recovers the performance loss, achieving 89.99% accuracy on VGG11 with 0.1% sparsification, compared to 88.81% in baseline compression.
• Distributed training over 16 devices reduces per-device memory usage from 243MB to 20.3MB and compute from 5.24GFlops to 0.33GFlops for ResNet-20 on CIFAR-10.
• The SCSP algorithm maintains high communication compression (up to 467×) while ensuring model accuracy remains within 1% of full-communication baselines.