[Paper Review] pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuning

Liping Yi, Han Yu+3 more|arXiv (Cornell University)|Oct 20, 2023

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes FedLoRA, a computation- and communication-efficient framework for model-heterogeneous personalized federated learning using low-rank adapters (LoRA). By training a shared, small low-rank adapter alongside each client’s heterogeneous local model and aggregating only the adapters on the server, FedLoRA enables bidirectional knowledge transfer, achieving 1.35% higher test accuracy than the best baseline, with 11.81× lower computation overhead and 7.41× lower communication cost.

ABSTRACT

Federated learning (FL) is an emerging machine learning paradigm in which a central server coordinates multiple participants (clients) collaboratively to train on decentralized data. In practice, FL often faces statistical, system, and model heterogeneities, which inspires the field of Model-Heterogeneous Personalized Federated Learning (MHPFL). With the increased interest in adopting large language models (LLMs) in FL, the existing MHPFL methods cannot achieve acceptable computational and communication costs, while maintaining satisfactory model performance. To bridge this gap, we propose a novel and efficient model-heterogeneous personalized Federated learning framework based on LoRA tuning (pFedLoRA). Inspired by the popular LoRA method for fine-tuning pre-trained LLMs with a low-rank model (a.k.a., an adapter), we design a homogeneous small adapter to facilitate federated client's heterogeneous local model training with our proposed iterative training for global-local knowledge exchange. The homogeneous small local adapters are aggregated on the FL server to generate a global adapter. We theoretically prove the convergence of pFedLoRA. Extensive experiments on two benchmark datasets demonstrate that pFedLoRA outperforms six state-of-the-art baselines, beating the best method by 1.35% in test accuracy, 11.81 times computation overhead reduction and 7.41 times communication cost saving.

Motivation & Objective

• Address the challenge of model heterogeneity in federated learning, where clients use different model architectures, especially in large language model (LLM) deployment.
• Overcome the high computational and communication costs of existing model-heterogeneous personalized FL methods, particularly when fine-tuning large models.
• Enable efficient knowledge sharing across clients with heterogeneous models while preserving model personalization and protecting intellectual property.
• Design a framework that maintains high model accuracy while drastically reducing training and communication overhead in real-world, resource-constrained FL settings.

Proposed method

• Introduce a homogeneous, low-rank adapter (LoRA) to each client’s heterogeneous local model, which is fine-tuned while the main model remains frozen.
• Use an iterative training process that enables bidirectional knowledge exchange between the global model (via aggregated adapters) and local models.
• Aggregate only the small, shared adapters on the server to form a global adapter, avoiding full model transmission and reducing communication costs.
• The final inference combines the frozen local model with the fine-tuned adapter, achieving performance comparable to full fine-tuning but with significantly reduced computation.
• Theoretical convergence analysis proves FedLoRA converges at a non-convex rate of 𝒪(1/T), supporting its stability and scalability.
• Leverage LoRA’s low-rank structure to maintain model efficiency, with only the adapter parameters updated during local training.

Experimental results

Research questions

• RQ1 Can a lightweight, shared adapter mechanism enable effective knowledge transfer in model-heterogeneous federated learning without requiring model homogeneity?
• RQ2 How does FedLoRA compare to state-of-the-art baselines in terms of test accuracy, computational efficiency, and communication cost under model heterogeneity?
• RQ3 To what extent does FedLoRA maintain robustness under varying degrees of data non-IIDness and client participation rates?
• RQ4 Does the use of LoRA-based adapters enable strong personalization while preserving global knowledge sharing in heterogeneous FL settings?

Key findings

• FedLoRA achieves 1.35% higher test accuracy than the best-performing baseline (FedProto) on benchmark datasets under model-heterogeneous settings.
• FedLoRA reduces computational overhead by up to 11.81× compared to FedProto, significantly improving training efficiency on resource-constrained clients.
• Communication costs are reduced by up to 7.41×, as only small adapter parameters are transmitted and aggregated, not full model weights.
• FedLoRA demonstrates superior robustness to data non-IIDness and varying client participation rates, consistently outperforming FedProto across all tested configurations.
• Visualization using T-SNE confirms that FedLoRA achieves better intra-class compactness and inter-class separation, indicating stronger personalization and generalization.
• Theoretical analysis confirms FedLoRA converges at a non-convex rate of 𝒪(1/T), validating its convergence under realistic FL conditions.