[Paper Review] Global Update Tracking: A Decentralized Learning Algorithm for Heterogeneous Data

Sai Aparna Aketi, Abolfazl Hashemi+1 more|arXiv (Cornell University)|May 8, 2023

Human Mobility and Location-Based Analysis4 citations

TL;DR

This paper proposes Global Update Tracking (GUT), a novel decentralized learning algorithm that mitigates performance degradation caused by heterogeneous data distributions without increasing communication overhead. By tracking model parameter updates rather than gradients, GUT achieves state-of-the-art test accuracy improvements of 1–6% over existing methods on CIFAR-10, CIFAR-100, Fashion MNIST, and ImageNette under non-IID data settings.

ABSTRACT

Decentralized learning enables the training of deep learning models over large distributed datasets generated at different locations, without the need for a central server. However, in practical scenarios, the data distribution across these devices can be significantly different, leading to a degradation in model performance. In this paper, we focus on designing a decentralized learning algorithm that is less susceptible to variations in data distribution across devices. We propose Global Update Tracking (GUT), a novel tracking-based method that aims to mitigate the impact of heterogeneous data in decentralized learning without introducing any communication overhead. We demonstrate the effectiveness of the proposed technique through an exhaustive set of experiments on various Computer Vision datasets (CIFAR-10, CIFAR-100, Fashion MNIST, and ImageNette), model architectures, and network topologies. Our experiments show that the proposed method achieves state-of-the-art performance for decentralized learning on heterogeneous data via a $1-6\%$ improvement in test accuracy compared to other existing techniques.

Motivation & Objective

• To address the performance degradation in decentralized learning caused by non-i.i.d. (heterogeneous) data distributions across devices.
• To develop a tracking-based method that improves model convergence and accuracy without increasing communication cost.
• To overcome the limitations of existing gradient-tracking or momentum-based approaches that incur 2× communication overhead.
• To enable effective decentralized training under real-world data heterogeneity while preserving privacy and scalability.
• To achieve convergence rates comparable to state-of-the-art centralized and decentralized algorithms under non-IID conditions.

Proposed method

• Proposes Global Update Tracking (GUT), which tracks the consensus of model parameter updates rather than gradients, reducing communication to only the tracking variable.
• Each agent stores local copies of neighbors’ model parameters and computes the difference (x_i^t - x_i^{t-1}) to track global updates.
• The algorithm uses a tracking variable y_i^t that is updated via local averaging and gradient descent, enabling consensus on model updates without transmitting full model parameters.
• Introduces a quasi-global momentum variant, QG-GUTm, which mimics global momentum synchronization without communication cost, improving convergence stability.
• Employs a scaling factor μ to control update magnitude, tuned per dataset and topology to balance convergence and performance.
• Theoretical analysis establishes a non-asymptotic convergence rate to a first-order solution, matching state-of-the-art decentralized algorithms.

Experimental results

Research questions

• RQ1 Can a tracking mechanism improve decentralized learning performance on heterogeneous data without increasing communication overhead?
• RQ2 How does tracking model parameter updates instead of gradients affect convergence and accuracy in decentralized settings?
• RQ3 Can a quasi-global momentum mechanism be effectively implemented in a decentralized, communication-free manner to mitigate data heterogeneity?
• RQ4 How does GUT compare to existing methods like DSGDm, QG-DSGDm, and D² in terms of test accuracy and convergence under varying degrees of data non-iidness?
• RQ5 What is the impact of different network topologies and model architectures on the performance of GUT and its variants?

Key findings

• QG-GUTm achieves a test accuracy of 81.04% ± 1.66 on CIFAR-10 with ResNet-20 under α=0.01 (high heterogeneity), outperforming QG-DSGDm by 1.19%.
• On 16-agent ring topology with ResNet-20, QG-GUTm achieves 88.22% ± 0.36 accuracy at α=1, surpassing DSGDm (86.60%) and GUT (84.72%).
• For CIFAR-100 with ResNet-20, QG-GUTm achieves 83.05% ± 0.48 at α=0.1, compared to DSGDm’s 74.20% ± 1.89, showing a 8.85% improvement.
• GUT alone achieves 70.16% ± 4.94 on CIFAR-10 with α=0.01, outperforming DSGDm (65.62%) and even the momentum variant DSGDm-N (63.66%).
• The use of Nesterov momentum in QG-GUTm-N reduces performance under high heterogeneity (α=0.01), indicating that quasi-global momentum is more effective than standard momentum in non-IID regimes.
• Experiments on VGG-11 and MobileNet-V2 confirm consistent gains across architectures, with QG-GUTm achieving 84.46% ± 0.33 on VGG-11 at α=1, compared to DSGDm’s 81.77%.