[Paper Review] AdapterFL: Adaptive Heterogeneous Federated Learning for Resource-constrained Mobile Computing Systems

Ruixuan Liu, Ming Hu+6 more|arXiv (Cornell University)|Nov 23, 2023

Privacy-Preserving Technologies in Data4 citations

TL;DR

AdapterFL proposes a novel heterogeneous federated learning framework that adaptively reassembles models by combining partial parameters from large and small models, enabling efficient training on resource-constrained mobile devices. It achieves up to 12% higher accuracy than state-of-the-art methods in heterogeneous, non-IID, and low-resource scenarios.

ABSTRACT

Federated Learning (FL) enables collaborative learning of large-scale distributed clients without data sharing. However, due to the disparity of computing resources among massive mobile computing devices, the performance of traditional homogeneous model-based Federated Learning (FL) is seriously limited. On the one hand, to achieve model training in all the diverse clients, mobile computing systems can only use small low-performance models for collaborative learning. On the other hand, devices with high computing resources cannot train a high-performance large model with their insufficient raw data. To address the resource-constrained problem in mobile computing systems, we present a novel heterogeneous FL approach named AdapterFL, which uses a model reassemble strategy to facilitate collaborative training of massive heterogeneous mobile devices adaptively. Specifically, we select multiple candidate heterogeneous models based on the computing performance of massive mobile devices and then divide each heterogeneous model into two partitions. By reassembling the partitions, we can generate models with varied sizes that are combined by the partial parameters of the large model with the partial parameters of the small model. Using these reassembled models for FL training, we can train the partial parameters of the large model using low-performance devices. In this way, we can alleviate performance degradation in large models due to resource constraints. The experimental results show that AdapterFL can achieve up to 12\% accuracy improvement compared to the state-of-the-art heterogeneous federated learning methods in resource-constrained scenarios.

Motivation & Objective

• Address the performance degradation in federated learning caused by device resource heterogeneity in mobile and WoT systems.
• Overcome the limitation of traditional FL, where global model size is constrained by the lowest-performance device.
• Enable high-performance devices to train larger models despite limited local data, while low-performance devices contribute effectively.
• Design a flexible, adaptive framework that dynamically selects and reassembles models based on device capabilities without requiring additional datasets.
• Improve model generalization and scalability under non-IID data and varying client resource ratios.

Proposed method

• Select multiple candidate heterogeneous models (small, medium, large) based on the computing capabilities of mobile devices.
• Partition each model into two functional components: a feature-extraction block and a classifier head.
• Reassemble models by combining the feature-extraction block from a large model with the classifier head from a small model, creating hybrid models of varying sizes.
• Train the reassembled models in a federated manner, allowing low-performance devices to train the classifier head while high-performance devices train the more complex feature-extraction block.
• Use a group-based strategy (L-group) to optimize model selection and aggregation, improving generalization and robustness.
• Leverage knowledge transfer across reassembled models through shared feature representations, minimizing performance loss due to resource constraints.

Experimental results

Research questions

• RQ1 How can federated learning be made more effective in mobile computing systems with highly heterogeneous device resources?
• RQ2 Can model reassembly from large and small models improve performance when training data is limited on high-end devices?
• RQ3 How does the proposed AdapterFL framework perform under non-IID data distributions and varying client resource ratios?
• RQ4 To what extent does the reassembly strategy enhance model accuracy compared to existing heterogeneous FL methods?
• RQ5 How scalable is AdapterFL under different numbers of clients and varying client activation ratios?

Key findings

• AdapterFL achieves up to 12% higher accuracy than state-of-the-art heterogeneous FL methods in resource-constrained scenarios.
• On the CIFAR-10 dataset, AdapterFL with the L-group configuration achieved 76.83% accuracy when 80% of clients were low-resource (small models), outperforming FedBase.
• In scenarios with 80% high-performance clients, AdapterFL achieved 79.32% accuracy with the L-M model, significantly outperforming FedBase’s 72.03%.
• AdapterFL maintains strong performance across varying client activation ratios: at 5% activation, it achieved 75.11% accuracy (L-S), compared to FedBase’s 73.49%.
• As the total number of clients increased from 50 to 500, AdapterFL maintained higher accuracy (e.g., 77.72% vs. 76.29% for L-S at N=50), demonstrating scalability.
• The L-group reassembly strategy consistently outperformed other configurations, proving that feature-extraction blocks from larger models enhance generalization and robustness.