[论文解读] Ditto: Fair and Robust Federated Learning Through Personalization

Tian Li, Shengyuan Hu+2 more|arXiv (Cornell University)|Dec 8, 2020

Privacy-Preserving Technologies in Data参考文献 73被引用 218

一句话总结

Ditto 引入一种全局正则化的多任务学习方法，用于联邦学习，个性化每个设备的模型，同时保留全局目标，从而在准确性、对数据/模型污染的鲁棒性以及设备间的公平性方面取得改进。

ABSTRACT

Fairness and robustness are two important concerns for federated learning systems. In this work, we identify that robustness to data and model poisoning attacks and fairness, measured as the uniformity of performance across devices, are competing constraints in statistically heterogeneous networks. To address these constraints, we propose employing a simple, general framework for personalized federated learning, Ditto, that can inherently provide fairness and robustness benefits, and develop a scalable solver for it. Theoretically, we analyze the ability of Ditto to achieve fairness and robustness simultaneously on a class of linear problems. Empirically, across a suite of federated datasets, we show that Ditto not only achieves competitive performance relative to recent personalization methods, but also enables more accurate, robust, and fair models relative to state-of-the-art fair or robust baselines.

研究动机与目标

• 解决在统计异质的联邦网络中公平性与鲁棒性之间的权衡。
• 提出一个简单而可扩展的个性化框架（Ditto），将个性化模型正则化为全局模型。
• 提供收敛性保证，并证明Ditto在准确性、鲁棒性和公平性方面达到或超过最先进基线。
• 分析个性化如何在多样数据集和攻击下本质上改善FL的鲁棒性和公平性。

提出的方法

• 将Ditto 表述为双层目标：对每个设备 k，最小化 F_k(v_k) + (lambda/2) ||v_k - w^*||^2，其中 w^* 是聚合目标 G(F_1(w), ..., F_K(w)) 的最优全局模型。
• 使用交替优化算法（算法1），服务器通过标准的FL求解器更新全局模型 w^t，每个设备通过不完全求解 min_v_k F_k(v_k) + (lambda/2)||v_k - w^t||^2 以本地更新更新其个性化模型 v_k。
• 通过允许对全局目标使用任何 G(·) 求解器（如 FedAvg）来保持模块性，保护隐私和通信属性。
• 给出收敛性保证：若 F_k 是强凸且光滑，且 w^t 收敛到 w^*，则 v_k^t 收敛到 v_k^*，并且具有相关的收敛速率（定理1）。
• 在一类线性问题上分析 Ditto 的公平/鲁棒性权衡，推导出最优的 lambda*，并展示 Pareto 收益。

实验结果

研究问题

• RQ1 在异质FL设置中，通过全球正则化的多任务学习的个性化联邦学习能否同时提高准确性、鲁棒性和公平性？
• RQ2 Ditto 的正则化参数 lambda 如何在本地个性化与全局一致性之间取得平衡，以提升公平性并抵御训练时攻击？
• RQ3 在常见的 FL 实践（设备参与有限、局部更新）下，Ditto 求解器表现出怎样的收敛性质？
• RQ4 Ditto 的公平性与鲁棒性收益是否能从凸设置推广到非凸模型和真实世界的FL基准？

主要发现

数据集	攻击级别	全局	本地	公平（TERM，t=1）	Ditto
Fashion MNIST	A1 clean	0.911 (0.08)	0.876 (0.10)	0.909 (0.07)	0.943 (0.06)
Fashion MNIST	A1 20% adversaries	0.897 (0.08)	0.874 (0.10)	0.751 (0.12)	0.944 (0.07)
Fashion MNIST	A1 50% adversaries	0.855 (0.10)	0.876 (0.11)	0.637 (0.13)	0.937 (0.07)
Fashion MNIST	A1 80% adversaries	0.753 (0.13)	0.879 (0.10)	0.547 (0.11)	0.907 (0.10)
Fashion MNIST	A2 20% adversaries	0.900 (0.08)	0.874 (0.10)	0.731 (0.13)	0.938 (0.07)
Fashion MNIST	A2 50% adversaries	0.882 (0.09)	0.876 (0.11)	0.637 (0.14)	0.930 (0.08)
Fashion MNIST	A2 80% adversaries	0.857 (0.10)	0.879 (0.10)	0.653 (0.13)	0.913 (0.09)
Fashion MNIST	A3 10% adversaries	0.753 (0.10)	0.874 (0.10)	0.601 (0.12)	0.921 (0.09)
Fashion MNIST	A3 20% adversaries	0.551 (0.13)	0.876 (0.10)	0.131 (0.16)	0.902 (0.09)
Fashion MNIST	A3 50% adversaries	0.275 (0.12)	0.874 (0.10)	0.131 (0.16)	0.873 (0.11)
FEMNIST	A1 clean	0.804 (0.11)	0.628 (0.15)	0.809 (0.11)	0.834 (0.09)
FEMNIST	A1 20% adversaries	0.773 (0.11)	0.620 (0.14)	0.636 (0.15)	0.802 (0.10)
FEMNIST	A1 50% adversaries	0.727 (0.12)	0.627 (0.14)	0.562 (0.13)	0.762 (0.11)
FEMNIST	A1 80% adversaries	0.574 (0.15)	0.607 (0.14)	0.478 (0.12)	0.672 (0.13)
FEMNIST	A2 10% adversaries	0.774 (0.11)	0.620 (0.14)	0.440 (0.15)	0.801 (0.09)
FEMNIST	A2 15% adversaries	0.703 (0.14)	0.627 (0.14)	0.336 (0.12)	0.700 (0.15)
FEMNIST	A2 20% adversaries	0.636 (0.15)	0.607 (0.14)	0.353 (0.12)	0.675 (0.14)
FEMNIST	A3 10% adversaries	0.517 (0.14)	0.607 (0.14)	0.316 (0.12)	0.685 (0.15)
FEMNIST	A3 15% adversaries	0.487 (0.14)	0.620 (0.14)	0.299 (0.11)	0.650 (0.14)
FEMNIST	A3 20% adversaries	0.314 (0.13)	0.620 (0.14)	0.316 (0.12)	0.613 (0.13)

• Ditto 在与个性化方法相比具有竞争力或更优的准确性，并在多样的FL数据集上优于若干鲁棒或公平基线。
• 在各种攻击和数据集上，Ditto 在平均意义上比最强鲁棒基线的测试准确率提高约6%（绝对值）。
• Ditto 将设备间测试准确率的方差降低约10%，显示公平性提升。
• Ditto 提供鲁棒性而不显著牺牲公平性，在某些情况下甚至在公平性和鲁棒性上同时优于最新基线。
• 对线性问题的理论分析确定了一个最优的 lambda*，在 Ditto 的空间内产生最准确、公平和鲁棒的解。