[Paper Review] FedSAE: A Novel Self-Adaptive Federated Learning Framework in Heterogeneous Systems

Li Li, Moming Duan+6 more|arXiv (Cornell University)|Apr 15, 2021

Privacy-Preserving Technologies in Data39 references4 citations

TL;DR

FedSAE proposes a self-adaptive federated learning framework that dynamically adjusts client workloads based on historical training performance and uses active learning to select high-value clients, significantly reducing stragglers and improving global model accuracy in heterogeneous systems. It achieves 26.7% higher test accuracy and 90.3% lower dropout rate compared to FedAvg on average.

ABSTRACT

Federated Learning (FL) is a novel distributed machine learning which allows thousands of edge devices to train model locally without uploading data concentrically to the server. But since real federated settings are resource-constrained, FL is encountered with systems heterogeneity which causes a lot of stragglers directly and then leads to significantly accuracy reduction indirectly. To solve the problems caused by systems heterogeneity, we introduce a novel self-adaptive federated framework FedSAE which adjusts the training task of devices automatically and selects participants actively to alleviate the performance degradation. In this work, we 1) propose FedSAE which leverages the complete information of devices' historical training tasks to predict the affordable training workloads for each device. In this way, FedSAE can estimate the reliability of each device and self-adaptively adjust the amount of training load per client in each round. 2) combine our framework with Active Learning to self-adaptively select participants. Then the framework accelerates the convergence of the global model. In our framework, the server evaluates devices' value of training based on their training loss. Then the server selects those clients with bigger value for the global model to reduce communication overhead. The experimental result indicates that in a highly heterogeneous system, FedSAE converges faster than FedAvg, the vanilla FL framework. Furthermore, FedSAE outperforms than FedAvg on several federated datasets - FedSAE improves test accuracy by 26.7% and reduces stragglers by 90.3% on average.

Motivation & Objective

• Address performance degradation in federated learning caused by systems heterogeneity, where device resource differences lead to stragglers and slow convergence.
• Develop a privacy-preserving method that avoids collecting sensitive hardware information by leveraging only public training history.
• Enable self-adaptive workload assignment per client based on historical training reliability and performance.
• Accelerate global model convergence by integrating active learning to select high-value clients for participation.
• Improve model accuracy and reduce communication overhead in real-world federated settings with dynamic client capabilities.

Proposed method

• Proposes FedSAE-Ira and FedSAE-Fassa, two self-adaptive algorithms that predict each client’s affordable training workload using historical training data.
• Uses a moving average-based workload prediction model that incorporates past training loss and convergence patterns to estimate client reliability.
• Introduces a dynamic client selection mechanism based on active learning, where the server selects clients with higher training value (lower loss) to improve global model quality.
• Employs a smooth index α = 0.95 and increment parameters γ₁ = 3, γ₂ = 1 to tune workload prediction accuracy and adaptability.
• Combines workload prediction with active learning to simultaneously reduce stragglers and accelerate convergence.
• Relies solely on public training history—no private device specifications—ensuring privacy preservation.

Experimental results

Research questions

• RQ1 How can federated learning systems dynamically adapt training workloads to heterogeneous clients without relying on private hardware information?
• RQ2 To what extent can workload prediction based on historical training performance reduce straggler rates in federated learning?
• RQ3 Can integrating active learning with workload adaptation accelerate global model convergence while maintaining high accuracy?
• RQ4 How does the proposed self-adaptive framework compare to FedAvg in terms of accuracy and training efficiency under high system heterogeneity?
• RQ5 What is the optimal balance between convergence speed and final model accuracy when using active learning in the client selection phase?

Key findings

• FedSAE improves absolute test accuracy by 26.7% on average compared to FedAvg across multiple federated datasets.
• The framework reduces the straggler rate by 90.3% on average, significantly mitigating performance degradation due to slow clients.
• FedSAE-Fassa outperforms FedSAE-Ira by reducing stragglers by 96.3%, due to better utilization of historical training patterns.
• Active learning integration accelerates convergence by at least 23.5% on average, with FedSAE-Ira+AL200 achieving the fastest convergence rate.
• The use of active learning slightly reduces final accuracy (by up to 2.3% on FEMNIST and 0.7% on MNIST), suggesting a trade-off between speed and precision.
• On FEMNIST, FedSAE-Ira+AL200 achieves 60% test accuracy in 48 rounds, compared to 30 rounds for FedAvg with E=15, demonstrating faster convergence.