[Paper Review] FLoRA: Enhancing Vision-Language Models with Parameter-Efficient Federated Learning

Duy Phuong Nguyen, J. Pablo Muñoz+1 more|arXiv (Cornell University)|Apr 12, 2024

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes FLoRA, a parameter-efficient federated learning framework that integrates Low-Rank Adaptation (LoRA) with vision-language models (VLMs), specifically fine-tuning the CLIP model’s text encoder in a decentralized setting. By training only LoRA adapters instead of full weights, FLoRA reduces communication overhead by up to 4,766× and achieves up to 30% higher accuracy than baseline federated methods, while cutting memory usage by 2.47× and accelerating training by 34.72×.

ABSTRACT

In the rapidly evolving field of artificial intelligence, multimodal models, e.g., integrating vision and language into visual-language models (VLMs), have become pivotal for many applications, ranging from image captioning to multimodal search engines. Among these models, the Contrastive Language-Image Pre-training (CLIP) model has demonstrated remarkable performance in understanding and generating nuanced relationships between text and images. However, the conventional training of such models often requires centralized aggregation of vast datasets, posing significant privacy and data governance challenges. To address these concerns, this paper proposes a novel approach that leverages Federated Learning and parameter-efficient adapters, i.e., Low-Rank Adaptation (LoRA), to train VLMs. This methodology preserves data privacy by training models across decentralized data sources and ensures model adaptability and efficiency through LoRA's parameter-efficient fine-tuning. Our approach accelerates training time by up to 34.72 times and requires 2.47 times less memory usage than full fine-tuning.

Motivation & Objective

• To address data privacy and communication inefficiencies in training vision-language models (VLMs) like CLIP using centralized data aggregation.
• To enable efficient, decentralized fine-tuning of VLMs through federated learning (FL) while preserving model performance.
• To explore parameter-efficient adaptation via LoRA in a federated setting, minimizing computational and memory costs.
• To evaluate the effectiveness of LoRA integration in different components of the CLIP model, particularly the text encoder.
• To identify optimal hyperparameters (e.g., rank, scaling factor) for LoRA in federated VLM training.

Proposed method

• Federated learning is used to train the CLIP model across decentralized clients, each holding local data, without sharing raw data.
• LoRA adapters are injected into the CLIP model’s text encoder projection layers to enable parameter-efficient fine-tuning.
• Only the LoRA adapter parameters are communicated between clients and server in each FL round, drastically reducing communication overhead.
• The global model is updated via FedAvg, where only the LoRA adapter weights are aggregated and redistributed.
• The LoRA adapter uses low-rank decomposition to update only a small subset of parameters, keeping the pre-trained weights frozen.
• Hyperparameters such as rank (r=2) and scaling factor (α) are tuned to balance performance and efficiency.

Experimental results

Research questions

• RQ1 How does LoRA-based fine-tuning in a federated setting compare to full fine-tuning in terms of accuracy and communication efficiency for VLMs?
• RQ2 Where in the CLIP architecture—text encoder, image encoder, or both—does LoRA adaptation yield the best performance-to-parameter ratio?
• RQ3 What is the optimal rank and scaling factor for LoRA adapters in federated vision-language learning under non-IID and few-shot conditions?
• RQ4 Can LoRA significantly reduce communication costs in federated VLM training without sacrificing model accuracy?
• RQ5 How does FLoRA perform across diverse data distributions, including IID, non-IID, and few-shot settings?

Key findings

• FLoRA reduces communication overhead by up to 4,766× compared to full fine-tuning, due to transmitting only LoRA adapter parameters.
• FLoRA achieves up to 30% higher accuracy than baseline federated learning methods across multiple benchmarks, including in non-IID and few-shot settings.
• Training with FLoRA accelerates convergence by up to 34.72× and reduces memory usage by 2.47× compared to full fine-tuning.
• A LoRA adapter with rank r=2 achieves optimal balance, reducing trainable parameters to 24,576 in the text encoder and 36,864 in the image encoder.
• LoRA integration in the text encoder outperforms image encoder adaptation, offering higher accuracy with fewer parameters.
• A moderate scaling factor (α) provides the best trade-off between performance gain and preservation of pre-trained features, avoiding overfitting or underfitting.