[论文解读] Towards Building the Federated GPT: Federated Instruction Tuning

Jianyi Zhang, Saeed Vahidian+7 more|arXiv (Cornell University)|May 9, 2023

Advanced Data Storage Technologies被引用 8

一句话总结

该论文提出 FedIT，一种基于参数高效微调（LoRA）的联邦学习框架，用于大语言模型（LLMs）的指令微调，实现隐私保护的分布式指令数据利用，并在本地数据有限的情况下展示了相较于集中训练的性能提升。

ABSTRACT

While "instruction-tuned" generative large language models (LLMs) have demonstrated an impressive ability to generalize to new tasks, the training phases heavily rely on large amounts of diverse and high-quality instruction data (such as ChatGPT and GPT-4). Unfortunately, acquiring high-quality data, especially when it comes to human-written data, can pose significant challenges both in terms of cost and accessibility. Moreover, concerns related to privacy can further limit access to such data, making the process of obtaining it a complex and nuanced undertaking. Consequently, this hinders the generality of the tuned models and may restrict their effectiveness in certain contexts. To tackle this issue, our study introduces a new approach called Federated Instruction Tuning (FedIT), which leverages federated learning (FL) as the learning framework for the instruction tuning of LLMs. This marks the first exploration of FL-based instruction tuning for LLMs. This is especially important since text data is predominantly generated by end users. Therefore, it is imperative to design and adapt FL approaches to effectively leverage these users' diverse instructions stored on local devices, while preserving privacy and ensuring data security. In the current paper, by conducting widely used GPT-4 auto-evaluation, we demonstrate that by exploiting the heterogeneous and diverse sets of instructions on the client's end with the proposed framework FedIT, we improved the performance of LLMs compared to centralized training with only limited local instructions. Further, in this paper, we developed a Github repository named Shepherd. This repository offers a foundational framework for exploring federated fine-tuning of LLMs using heterogeneous instructions across diverse categories.

研究动机与目标

• Highlight the challenges of acquiring large, high-quality, and privacy-preserving instruction data for LLM instruction tuning.
• Introduce Federated Instruction Tuning (FedIT) as a privacy-preserving FL approach using parameter-efficient fine-tuning.
• Show that diverse, heterogeneous client instructions can improve model generalization in FL settings.
• Provide a reusable framework ( Shepherd ) to facilitate Federated PEIT research and experiments.

提出的方法

• Propose FedIT: clients initialize with a global LLM and a trainable LoRA adapter; clients perform local instruction tuning on their data without sharing raw data.
• Use parameter-efficient fine-tuning (LoRA) to update adapters instead of full model parameters, reducing compute and communication.
• Server aggregates local adapters via FedAvg and iterates with client selection to handle heterogeneity and resource variability.
• Demonstrate FedIT on a 7B LLaMA-based setup with 10 clients, 20 rounds, and 5% participation per round, using LoRA rank 8.
• Release Shepherd, a GitHub framework that supports client data allocation, scheduling, local training, and model aggregation for FedIT experiments.

实验结果

研究问题

• RQ1 Can federated instruction tuning (FedIT) achieve competitive performance compared to centralized instruction tuning when data is decentralized and privacy is preserved?
• RQ2 How does instruction- and language/domain heterogeneity across clients affect FedIT performance?
• RQ3 What are the trade-offs between communication/computation cost and model performance when using LoRA in federated instruction tuning?
• RQ4 Can a reusable framework like Shepherd facilitate scalable, privacy-preserving instruction tuning research across diverse LLMs and datasets?

主要发现

Baseline	Task	Scores	Relative Score
CentralizedModel	Centralized tuning with all the instructions	(142.2 ,130.7)	0.919
LLaMA	No instruction tuning	(114.0,131.7)	1.155
Local-1	Brainstorming instruction tuning	(120.0,131.0)	1.092
Local-2	Closed question answering instruction tuning	(116.1,129.0)	1.111
Local-3	Classification and brainstorming instruction tuning	(121.3,131.8)	1.087

• FedIT with heterogeneous client instructions yields higher GPT-4 auto-evaluation scores than LLaMA baseline and local-only training.
• FedIT outperforms centralized training with only limited local instructions but trails the centralized model trained on full data, highlighting room for optimization in federated aggregation.
• Heterogeneity in instructions across languages and domains can be beneficial for generalization in FedIT, given appropriate federated optimization strategies.
• LoRA-based parameter-efficient tuning dramatically reduces trainable parameters and communication costs, enabling scalable federated fine-tuning on edge devices.
• A public framework Shepherd is released to support Federated PEIT research and experimentation.