[Paper Review] Convergence Analysis of Sequential Federated Learning on Heterogeneous Data

Yipeng Li, Xinchen Lyu|arXiv (Cornell University)|Nov 6, 2023

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper establishes the first convergence analysis for sequential federated learning (SFL) on heterogeneous data, proving SFL achieves better convergence than parallel FL (PFL) under strong, general, and non-convex objectives. The analysis reveals SFL outperforms PFL in cross-device settings with extreme data heterogeneity, validated by experiments showing up to 10% higher test accuracy on CIFAR-10 and CINIC-10 under extreme client drift.

ABSTRACT

There are two categories of methods in Federated Learning (FL) for joint training across multiple clients: i) parallel FL (PFL), where clients train models in a parallel manner; and ii) sequential FL (SFL), where clients train models in a sequential manner. In contrast to that of PFL, the convergence theory of SFL on heterogeneous data is still lacking. In this paper, we establish the convergence guarantees of SFL for strongly/general/non-convex objectives on heterogeneous data. The convergence guarantees of SFL are better than that of PFL on heterogeneous data with both full and partial client participation. Experimental results validate the counterintuitive analysis result that SFL outperforms PFL on extremely heterogeneous data in cross-device settings.

Motivation & Objective

• To close the gap in theoretical understanding by establishing convergence guarantees for sequential federated learning (SFL) on heterogeneous data.
• To compare the convergence performance of SFL and parallel FL (PFL) under strong, general, and non-convex objectives with data heterogeneity.
• To provide theoretical justification for the empirical observation that SFL can outperform PFL in extreme data heterogeneity scenarios.
• To extend the theoretical framework to apply to split learning (SL), demonstrating broader applicability of the convergence results.

Proposed method

• Proposes a novel convergence analysis framework for SFL under heterogeneous data, covering strongly convex, general convex, and non-convex objectives.
• Uses a sequential update rule where each client initializes with the latest parameters from the previous client, performing K local SGD steps before passing parameters forward.
• Derives convergence bounds using assumptions on gradient Lipschitz continuity, bounded variance, and client data heterogeneity (measured via divergence between local and global gradients).
• Applies the analysis to both full and partial client participation, showing tighter convergence rates for SFL compared to PFL.
• Demonstrates that the theoretical framework is also applicable to split learning (SL), extending its relevance to edge AI systems.
• Employs a weighted averaging strategy for final model aggregation, ensuring convergence to a stationary point under the derived bounds.

Experimental results

Research questions

• RQ1 Does sequential FL (SFL) achieve better convergence than parallel FL (PFL) on heterogeneous data under strongly convex objectives?
• RQ2 How does SFL perform compared to PFL under general and non-convex objectives with data heterogeneity?
• RQ3 What is the theoretical convergence rate of SFL under full and partial client participation in heterogeneous settings?
• RQ4 Can the convergence theory of SFL be extended to other distributed learning paradigms like split learning (SL)?
• RQ5 Why does SFL outperform PFL in cross-device FL with extreme data heterogeneity, despite the absence of explicit theoretical justification in prior work?

Key findings

• SFL achieves better convergence guarantees than PFL on heterogeneous data across all objective types: strongly convex, general convex, and non-convex.
• In cross-device settings with extreme data heterogeneity, SFL outperforms PFL, with test accuracy gains of up to 10% on CIFAR-10 and CINIC-10 under full client participation.
• On MNIST and FMNIST, SFL achieves higher test accuracy than PFL in all settings, with MLP and LeNet-5 models showing 1–2% improvements under high heterogeneity.
• For VGG-9 on CIFAR-10, SFL achieves 82.56% test accuracy (vs. 78.42% for PFL) with K=5 and C=2, demonstrating a 4.14% improvement under partial participation.
• On CINIC-10 with ResNet-18, SFL achieves 64.17% test accuracy (vs. 57.70% for PFL) under C=2 and K=5, a 6.47% improvement, confirming superior performance in extreme heterogeneity.
• The convergence analysis confirms that SFL reduces client drift more effectively than PFL due to sequential parameter propagation, which stabilizes training under non-iid data.