[Paper Review] Intermittent Pulling with Local Compensation for Communication-Efficient Federated Learning

Haozhao Wang, Zhihao Qu+4 more|arXiv (Cornell University)|Jan 22, 2020

Privacy-Preserving Technologies in Data13 references4 citations

TL;DR

This paper proposes Pulling Reduction with Local Compensation (PRLC), a communication-efficient federated learning method where workers intermittently pull the global model and use local updates to compensate for synchronization gaps. PRLC achieves up to 50% reduction in pulling operations compared to state-of-the-art methods like LAG, while maintaining convergence rates comparable to non-reduction methods.

ABSTRACT

Federated Learning is a powerful machine learning paradigm to cooperatively train a global model with highly distributed data. A major bottleneck on the performance of distributed Stochastic Gradient Descent (SGD) algorithm for large-scale Federated Learning is the communication overhead on pushing local gradients and pulling global model. In this paper, to reduce the communication complexity of Federated Learning, a novel approach named Pulling Reduction with Local Compensation (PRLC) is proposed. Specifically, each training node intermittently pulls the global model from the server in SGD iterations, resulting in that it is sometimes unsynchronized with the server. In such a case, it will use its local update to compensate the gap between the local model and the global model. Our rigorous theoretical analysis of PRLC achieves two important findings. First, we prove that the convergence rate of PRLC preserves the same order as the classical synchronous SGD for both strongly-convex and non-convex cases with good scalability due to the linear speedup with respect to the number of training nodes. Second, we show that PRLC admits lower pulling frequency than the existing pulling reduction method without local compensation. We also conduct extensive experiments on various machine learning models to validate our theoretical results. Experimental results show that our approach achieves a significant pulling reduction over the state-of-the-art methods, e.g., PRLC requiring only half of the pulling operations of LAG.

Motivation & Objective

• Address the high communication overhead in federated learning caused by frequent model pulling and gradient pushing.
• Reduce the number of pulling operations without sacrificing convergence speed or model accuracy.
• Improve scalability and efficiency in large-scale, decentralized federated learning systems with limited bandwidth.
• Develop a theoretically grounded method that maintains convergence guarantees under intermittent synchronization.

Proposed method

• Workers in each SGD iteration randomly decide whether to pull the global model from the server, introducing intermittent synchronization.
• When not pulling, workers use their local model updates to approximate the global update and compensate for the model gap.
• The local update acts as a proxy for the average update across all workers, reducing reliance on frequent server pulls.
• Theoretical analysis proves convergence rates of PRLC match classical synchronous SGD in both strongly convex and non-convex settings.
• A pulling ratio $ r $ controls the frequency of pulls, with $ r = 0.4 $ used in experiments to balance communication cost and convergence.
• The method is designed to be robust under non-IID data and low-bandwidth edge environments, validated via real-world cluster simulations.

Experimental results

Research questions

• RQ1 Can intermittent pulling with local compensation maintain convergence rates comparable to synchronous SGD in federated learning?
• RQ2 How does local compensation improve performance compared to pulling reduction without compensation?
• RQ3 What is the minimum pulling frequency (i.e., pulling ratio) that still ensures convergence and model accuracy?
• RQ4 How does PRLC compare to ASGD and LAG in terms of convergence time and communication efficiency?
• RQ5 Does PRLC scale linearly with the number of training nodes, and how does it perform under non-IID data distributions?

Key findings

• PRLC achieves a 50% reduction in pulling operations compared to LAG, requiring only half the number of pulls to reach convergence on both Logistic Regression and ResNet18 models.
• The convergence rate of PRLC is in the same order as classical synchronous SGD for both strongly convex and non-convex objectives.
• PRLC maintains better scalability than ASGD, with linear speedup in convergence time as the number of training nodes increases.
• Experiments show PRLC converges approximately 30% faster than ASGD in low-bandwidth edge environments, with convergence times of ~100,000s versus ~140,000s.
• Local compensation significantly improves performance: PRLC outperforms the baseline method without compensation (PR) by reducing convergence loss gaps, especially in early training stages.
• The convergence gap in PRLC decreases quadratically with learning rate decay, confirming theoretical predictions about second-order convergence improvement.