[Paper Review] FLGo: A Fully Customizable Federated Learning Platform

Zheng Wang, Xiaoliang Fan+7 more|arXiv (Cornell University)|Jun 21, 2023

Privacy-Preserving Technologies in Data4 citations

TL;DR

FLGo is a lightweight, fully customizable federated learning platform that enables high-shareability through a plugin-based architecture, offering 40+ benchmarks, 20+ algorithms, and 2 system simulators as out-of-the-box plugins. It supports customizable data and system heterogeneity, asynchronous training, and experimental tools like parallel acceleration and hyperparameter auto-tuning, significantly reducing customization complexity for cross-application FL research.

ABSTRACT

Federated learning (FL) has found numerous applications in healthcare, finance, and IoT scenarios. Many existing FL frameworks offer a range of benchmarks to evaluate the performance of FL under realistic conditions. However, the process of customizing simulations to accommodate application-specific settings, data heterogeneity, and system heterogeneity typically remains unnecessarily complicated. This creates significant hurdles for traditional ML researchers in exploring the usage of FL, while also compromising the shareability of codes across FL frameworks. To address this issue, we propose a novel lightweight FL platform called FLGo, to facilitate cross-application FL studies with a high degree of shareability. Our platform offers 40+ benchmarks, 20+ algorithms, and 2 system simulators as out-of-the-box plugins. We also provide user-friendly APIs for quickly customizing new plugins that can be readily shared and reused for improved reproducibility. Finally, we develop a range of experimental tools, including parallel acceleration, experiment tracker and analyzer, and parameters auto-tuning. FLGo is maintained at \url{flgo-xmu.github.io}.

Motivation & Objective

• Address the high complexity and low code shareability in customizing federated learning (FL) simulations for application-specific settings, data heterogeneity, and system heterogeneity.
• Overcome the limitations of existing FL frameworks that lack standardized, reusable customization mechanisms despite offering benchmarks and algorithms.
• Enable researchers to easily develop, share, and reuse custom plugins for benchmarks, algorithms, and system simulators to improve reproducibility and cross-framework collaboration.
• Provide a unified experimental environment with tools for parallel execution, experiment tracking, and hyperparameter tuning to streamline FL evaluation.

Proposed method

• Implement benchmarks, algorithms, and system simulators as independent, pluggable components to decouple functionality and enhance reusability.
• Design a plugin library that allows developers to distribute and share locally customized plugins, promoting code reuse and community-driven extension.
• Integrate 40+ standardized benchmarks covering diverse datasets (e.g., CIFAR-10, Shakespeare, Reddit) and data distribution patterns (IID, Dirichlet, Louvain).
• Support 20+ FL algorithms including FedAvg, FedProx, Scaffold, and FedNova, with configurable training and aggregation strategies.
• Provide two system simulators to model client-side heterogeneity: client availability, completeness, connectivity, and responsiveness.
• Include experimental tools such as parallel acceleration, experiment tracker and analyzer, and automated hyperparameter tuning to improve efficiency and reproducibility.

Figure 1: FLGo implements benchmarks, simulators, and algorithms as independent plugins, where developers can download plugins and share locally customized plugins through a plugin library.

Experimental results

Research questions

• RQ1 Can a federated learning platform be designed to support high customization while maintaining high code shareability across diverse application scenarios?
• RQ2 How effectively can a plugin-based architecture reduce the complexity of simulating data and system heterogeneity in FL experiments?
• RQ3 To what extent do different FL algorithms perform under various data and system heterogeneity configurations, and can the platform support such comparative evaluations?
• RQ4 Can asynchronous training strategies significantly reduce training time costs under complex, realistic heterogeneity combinations compared to synchronous methods?
• RQ5 How does the platform’s extensibility and tooling support reproducibility and scalability in cross-application FL research?

Key findings

• No single FL algorithm consistently outperforms others across all datasets and heterogeneity settings; FedAvg performs best on the Louvain distribution in Cora, while Scaffold excels on IID data.
• Non-IID data distributions consistently degrade model performance across all algorithms compared to IID settings, with performance drops observed in both FashionMNIST and Reddit datasets.
• Client availability heterogeneity has a non-trivial negative impact on model performance, with the worst performance observed when data and availability heterogeneity co-occur.
• Client completeness heterogeneity reduces training efficiency, and although FedNova is designed to be robust, it underperforms FedAvg and FedProx in FashionMNIST due to over-enlarged short model updates.
• Reduced client connectivity increases training time costs, especially in non-IID settings on Reddit, where performance degrades significantly.
• Asynchronous training (FedAsync) reduces time costs compared to synchronous methods under complex heterogeneity, but achieves worse model performance than FedAvg and FedProx after the same number of aggregations in non-IID scenarios.

Figure 2: The main workflow in FLGo.