[论文解读] Efficient Personalized Federated Learning via Sparse Model-Adaptation

Daoyuan Chen, Liuyi Yao+3 more|arXiv (Cornell University)|May 4, 2023

Privacy-Preserving Technologies in Data被引用 9

一句话总结

本文介绍了 pFedGate，一种个性化联邦学习方法，通过一个轻量级门控层学习稀疏的本地模型，在异构数据和资源约束下实现个性化与效率，并具有理论保证和强大的经验结果。

ABSTRACT

Federated Learning (FL) aims to train machine learning models for multiple clients without sharing their own private data. Due to the heterogeneity of clients' local data distribution, recent studies explore the personalized FL that learns and deploys distinct local models with the help of auxiliary global models. However, the clients can be heterogeneous in terms of not only local data distribution, but also their computation and communication resources. The capacity and efficiency of personalized models are restricted by the lowest-resource clients, leading to sub-optimal performance and limited practicality of personalized FL. To overcome these challenges, we propose a novel approach named pFedGate for efficient personalized FL by adaptively and efficiently learning sparse local models. With a lightweight trainable gating layer, pFedGate enables clients to reach their full potential in model capacity by generating different sparse models accounting for both the heterogeneous data distributions and resource constraints. Meanwhile, the computation and communication efficiency are both improved thanks to the adaptability between the model sparsity and clients' resources. Further, we theoretically show that the proposed pFedGate has superior complexity with guaranteed convergence and generalization error. Extensive experiments show that pFedGate achieves superior global accuracy, individual accuracy and efficiency simultaneously over state-of-the-art methods. We also demonstrate that pFedGate performs better than competitors in the novel clients participation and partial clients participation scenarios, and can learn meaningful sparse local models adapted to different data distributions.

研究动机与目标

• 解决个性化联邦学习（PFL）中的资源与计算异质性问题。
• 学习能够适应本地数据分布、同时尊重客户端资源限制的个性化稀疏模型。
• 为稀疏门控PFL框架的收敛性与泛化性提供理论保证。
• 在标准 FL 基准上展示更优的全局与局部（每个客户端）准确性与效率。

提出的方法

• 为每个客户端引入一个轻量级的可训练门控层，以在本地数据批次条件下产生基于批次的、按块稀疏的权重。
• 通过逐元素门控将共享全局模型转变为个性化稀疏模型：h_theta'i = h_theta_g ∘ M'i，由 M'i 由 g_{phi_i}(x) 在稀疏性约束下生成。
• 使用基于背包的预测块权重调整来强制硬稀疏性，以满足客户端特定的稀疏性上限。
• 支持无算子类型的块划分，以便灵活将模型划分为子块以进行剪枝/自适应。
• 给出时空分析，表明与密集基线相比，由于稀疏性和门控，计算和通信成本降低。
• 在有界多样性和标准 FL 假设下给出泛化与收敛性结果。

实验结果

研究问题

• RQ1 在客户端数据和资源各异的情况下，如何在 FL 中高效实现个性化？
• RQ2 在不牺牲收敛性或泛化性的前提下，稀疏的、带门控的对共享全局模型的适配是否能满足单个客户端的需求？
• RQ3 对于稀疏、带门控的个性化 FL，理论保证（泛化、收敛、复杂度）是什么？
• RQ4 与最先进的 PFL 方法相比，pFedGate 在标准、部分参与和新客户端场景下的表现如何？

主要发现

• pFedGate 在显著降低稀疏性的同时实现全球与个体准确性优越提升（例如在最强竞争的 PFL 方法下，平均准确性提升高达 4.53%，且稀疏度降低 12 倍）。
• 该方法通过联合学习稀疏本地模型和门控权重，提高计算和通信成本相对于基线的效率。
• pFedGate 在部分客户端参与和新客户端参与设置下表现出鲁棒性，学习出与数据分布相关的有意义的稀疏本地模型。
• 理论结果对收敛与泛化提供保证，且时空复杂度分析显示相对最先进方法的优势。