[Paper Review] Server Averaging for Federated Learning

George Pu, Yanlin Zhou+2 more|arXiv (Cornell University)|Mar 22, 2021

Privacy-Preserving Technologies in Data16 references4 citations

TL;DR

This paper proposes server averaging for federated learning, where the global model is periodically averaged from multiple past global models to accelerate convergence. Experiments show it reaches target accuracy faster than FedAvg, especially on non-IID data, while epoch decay reduces client computation by up to 40% without sacrificing performance.

ABSTRACT

Federated learning allows distributed devices to collectively train a model without sharing or disclosing the local dataset with a central server. The global model is optimized by training and averaging the model parameters of all local participants. However, the improved privacy of federated learning also introduces challenges including higher computation and communication costs. In particular, federated learning converges slower than centralized training. We propose the server averaging algorithm to accelerate convergence. Sever averaging constructs the shared global model by periodically averaging a set of previous global models. Our experiments indicate that server averaging not only converges faster, to a target accuracy, than federated averaging (FedAvg), but also reduces the computation costs on the client-level through epoch decay.

Motivation & Objective

• To address the slow convergence of federated learning, especially under non-i.i.d. data distributions.
• To reduce client-side computation costs without degrading model performance.
• To improve convergence speed and generalization by leveraging historical global model states through server-side averaging.
• To explore the effectiveness of epoch decay as a client-level computation control mechanism.

Proposed method

• Server averaging periodically computes a new global model by averaging the past P global models, using a sliding window of size P.
• The averaging is performed every R communication rounds, with the formula $ w^t ext{←} \sum_{i=0}^{P-1} w^{t-i} $, effectively creating a weighted average of recent global models.
• Epoch decay reduces the number of local training epochs over time, starting from an initial E and halving every D rounds: $ E_t = \max(E / 2^{\lfloor t/D \rfloor}, 1) $.
• The method is applied to non-IID federated MNIST and evaluated against FedAvg using test accuracy at 90%, 95%, 97%, and 98% thresholds.
• The approach leverages the principle of iterate averaging to stabilize training and improve generalization, similar to Stochastic Weight Averaging (SWA).
• The technique is designed to be communication- and computation-efficient, avoiding the need to transmit additional gradients or momentum terms.

Experimental results

Research questions

• RQ1 Can periodic averaging of past global models on the server side accelerate convergence in federated learning compared to FedAvg?
• RQ2 How does server averaging affect convergence speed and model accuracy under non-i.i.d. data distributions?
• RQ3 Can epoch decay reduce client-side computation while maintaining or improving convergence performance?
• RQ4 Does combining server averaging with epoch decay lead to a better trade-off between training speed and computational cost?
• RQ5 How does the performance of server averaging compare to state-of-the-art methods like SCAFFOLD or FedProx in terms of convergence and communication efficiency?

Key findings

• Server averaging with P=2 and R=40 reduced the time to reach 90% accuracy by 23% compared to FedAvg (28.00 vs. 36.40 rounds).
• For 95% accuracy, the best configuration (P=3, R=40) achieved 76.20 rounds, a 11.5% improvement over FedAvg’s 85.40 rounds.
• The method achieved faster convergence across all accuracy thresholds, with the most significant gains at 90% and 95% accuracy.
• Epoch decay reduced client computation by up to 40%, with the best decay rate (D=200) achieving 27.20 rounds for 90% accuracy, a 25% improvement over FedAvg.
• Despite reduced computation, epoch decay maintained competitive performance, with only minor increases in rounds needed for higher accuracy thresholds.
• The combination of server averaging and epoch decay outperformed FedAvg in convergence speed while reducing client computation, demonstrating a favorable trade-off between speed and cost.