[论文解读] Personalized Federated Learning with Clustered Generalization

Xueyang Tang, Song Guo+1 more|arXiv (Cornell University)|Jun 24, 2021

Privacy-Preserving Technologies in Data参考文献 50被引用 10

一句话总结

本文提出聚类泛化个性化联邦学习（CG-PFL），通过维护多个全局模型来捕捉多样化的数据簇，并指导个性化本地模型。通过将PFL建模为双层优化问题，CG-PFL在合成数据集和真实世界数据集上均优于SOTA方法，实现了更优的泛化与个性化权衡。

ABSTRACT

We study the recent emerging personalized federated learning (PFL) that aims at dealing with the challenging problem of Non-I.I.D. data in the federated learning (FL) setting. The key difference between PFL and conventional FL lies in the training target, of which the personalized models in PFL usually pursue a trade-off between personalization (i.e., usually from local models) and generalization (i.e., usually from the global model) on trained models. Conventional FL methods can hardly meet this target because of their both well-developed global and local models. The prevalent PFL approaches usually maintain a global model to guide the training process of local models and transfer a proper degree of generalization to them. However, the sole global model can only provide one direction of generalization and may even transfer negative effects to some local models when rich statistical diversity exists across multiple local datasets. Based on our observation, most real or synthetic data distributions usually tend to be clustered to some degree, of which we argue different directions of generalization can facilitate the PFL. In this paper, we propose a novel concept called clustered generalization to handle the challenge of statistical heterogeneity in FL. Specifically, we maintain multiple global (generalized) models in the server to associate with the corresponding amount of local model clusters in clients, and further formulate the PFL as a bi-level optimization problem that can be solved efficiently and robustly. We also conduct detailed theoretical analysis and provide the convergence guarantee for the smooth non-convex objectives. Experimental results on both synthetic and real datasets show that our approach surpasses the state-of-the-art by a significant margin.

研究动机与目标

• 为解决由于客户端间数据分布非独立同分布（non-IID）导致的个性化联邦学习（PFL）中统计异质性的挑战。
• 克服传统PFL中单一全局模型的局限性，该模型可能将负面泛化效应传递给部分客户端。
• 探究多个、针对特定簇的全局模型是否能更好地捕捉多样化数据分布并提升个性化程度。
• 设计一个稳健且高效的双层优化框架，支持多个全局模型与本地簇的协同优化。
• 为所提出的框架在平滑非凸目标函数下的收敛性提供理论保证。

提出的方法

• 该方法在服务器端引入多个全局模型，每个模型与一组本地客户端模型所属的特定簇相关联。
• 根据本地数据分布的相似性，将客户端分组为若干簇。
• 构建双层优化框架：上层优化多个全局模型，下层为每个簇内的本地模型进行适应性调整。
• 通过一种高效的交替算法求解优化问题，交替更新全局模型与本地模型。
• 该方法利用簇感知的泛化方向，在保持对多样化数据分布鲁棒性的同时提升个性化程度。
• 理论分析为所提出的双层框架在平滑非凸目标函数下的收敛性提供了保证。

实验结果

研究问题

• RQ1 在非独立同分布（non-IID）数据条件下，每个与客户端簇对齐的多个全局模型是否能提升联邦学习中的个性化性能？
• RQ2 与单一全局模型方法相比，聚类泛化在处理统计异质性方面表现如何？
• RQ3 所提出的双层优化框架在非凸设置下的收敛行为如何？
• RQ4 该方法是否能在多样化的合成数据集与真实世界数据分布上保持稳健性能？
• RQ5 基于聚类的泛化是否能减少传统PFL中观察到的负迁移效应？

主要发现

• 所提出的CG-PFL方法在合成数据集与真实世界数据集上显著优于当前最先进的个性化联邦学习方法。
• 使用多个针对簇的全局模型相比单一全局模型基线，实现了更优的泛化与个性化权衡。
• 该方法在客户端间存在高度数据异质性与统计多样性时仍表现出稳健性能。
• 理论分析证实了所提双层优化框架在平滑非凸目标函数下的收敛性。
• 实验结果表明，簇感知的泛化机制可有效减少负迁移效应，并提升客户端特定模型的准确率。
• 与标准PFL基线相比，该方法在仅增加极少计算开销的前提下实现了性能提升。