[Paper Review] Federated Adaptive Prompt Tuning for Multi-Domain Collaborative Learning

Shangchao Su, Mingzhao Yang+2 more|arXiv (Cornell University)|Nov 15, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes FedAPT, a federated adaptive prompt tuning method for multi-domain image classification using CLIP, where clients collaboratively train an adaptive network and meta prompt with frozen domain-specific keys. By dynamically selecting personalized prompts per test sample via a learned adaptive network, FedAPT achieves performance exceeding fully fine-tuned models with under 10% of the parameters, demonstrating strong generalization across diverse client domains in both supervised and unsupervised settings.

ABSTRACT

Federated learning (FL) enables multiple clients to collaboratively train a global model without disclosing their data. Previous researches often require training the complete model parameters. However, the emergence of powerful pre-trained models makes it possible to achieve higher performance with fewer learnable parameters in FL. In this paper, we propose a federated adaptive prompt tuning algorithm, FedAPT, for multi-domain collaborative image classification with powerful foundation models, like CLIP. Compared with direct federated prompt tuning, our core idea is to adaptively unlock specific domain knowledge for each test sample in order to provide them with personalized prompts. To implement this idea, we design an adaptive prompt tuning module, which consists of a meta prompt, an adaptive network, and some keys. The server randomly generates a set of keys and assigns a unique key to each client. Then all clients cooperatively train the global adaptive network and meta prompt with the local datasets and the frozen keys. Ultimately, the global aggregation model can assign a personalized prompt to CLIP based on the domain features of each test sample. We perform extensive experiments on two multi-domain image classification datasets across two different settings -- supervised and unsupervised. The results show that FedAPT can achieve better performance with less than 10\% of the number of parameters of the fully trained model, and the global model can perform well in diverse client domains simultaneously. The source code is available at \url{https://github.com/leondada/FedAPT}.

Motivation & Objective

• To address the challenge of cross-domain performance degradation in federated learning due to non-IID data across diverse client domains.
• To reduce communication and training costs in federated learning by tuning only a small number of parameters, leveraging powerful pre-trained models like CLIP.
• To enable personalized prompt generation for each test sample based on its domain-specific features, improving classification accuracy across all domains simultaneously.
• To develop a robust, privacy-preserving framework that maintains high performance in both supervised and unsupervised settings without sharing raw data.

Proposed method

• Clients are assigned unique frozen keys by the server, which remain fixed during training and serve as domain-specific embeddings.
• An adaptive network learns to select the most relevant key for each test image based on its domain features, enabling dynamic prompt generation.
• The meta prompt is shared globally and combined with the selected key via element-wise multiplication to produce a personalized prompt for each input.
• The adaptive network and meta prompt are jointly trained via federated learning using local data, with only the network and prompt parameters updated and communicated.
• A temperature hyperparameter controls the softness of key selection, balancing performance on seen domains and generalization to unseen domains.
• During inference, the adaptive network selects a key per sample, and the corresponding prompt is applied to the frozen CLIP model for classification.

Experimental results

Research questions

• RQ1 Can adaptive prompt tuning with domain-specific keys improve performance in multi-domain federated image classification compared to fixed or global prompts?
• RQ2 Does the use of a lightweight adaptive network and meta prompt allow FedAPT to outperform fully fine-tuned models with significantly fewer learnable parameters?
• RQ3 How does the adaptive key selection mechanism affect performance on both seen and unseen client domains?
• RQ4 Can FedAPT maintain strong performance in unsupervised settings where labels are unavailable?
• RQ5 Does the inclusion of frozen keys during local training hinder local model optimization or generalization?

Key findings

• FedAPT achieves a global model accuracy of 67.39% on the Office-Caltech10 dataset in the supervised setting, surpassing fully fine-tuned ResNet50 and ViT models with less than 10% of their parameters.
• In the unsupervised setting, FedAPT achieves 68.26% accuracy on the same dataset, outperforming PromptFL (67.26%) and demonstrating strong zero-shot generalization.
• The adaptive network with a temperature of τ=0.01 improves performance on seen domains (e.g., 72.76% on domain i) while maintaining nearly unchanged generalization on unseen domains.
• Using domain-specific keys significantly boosts performance in their respective domains—e.g., the k-th key improves accuracy in the k-th domain by up to 15 percentage points compared to using the meta prompt alone.
• The use of keys does not limit local model training, as shown by comparable local model performance with and without keys in ablation studies.
• Inference efficiency is dramatically improved: inference time is reduced from 30 minutes to 30 seconds on a single GPU, with only a minor increase in GFLOPs (4.42 for FedAPT vs. 4.14 for ResNet50).