[論文レビュー] Ditto: Fair and Robust Federated Learning Through Personalization

Tian Li, Shengyuan Hu+2 more|arXiv (Cornell University)|Dec 8, 2020

Privacy-Preserving Technologies in Data参考文献 73被引用数 218

ひとこと要約

Dittoは、グローバルな正則化を用いたマルチタスク学習アプローチをフェデレーテッド学習に導入し、デバイスごとのモデルを個別化しつつグローバルな目的を保持します。これにより、精度の向上、データ汚染およびモデル汚染に対する頑健性、デバイス間の公平性が向上します。

ABSTRACT

Fairness and robustness are two important concerns for federated learning systems. In this work, we identify that robustness to data and model poisoning attacks and fairness, measured as the uniformity of performance across devices, are competing constraints in statistically heterogeneous networks. To address these constraints, we propose employing a simple, general framework for personalized federated learning, Ditto, that can inherently provide fairness and robustness benefits, and develop a scalable solver for it. Theoretically, we analyze the ability of Ditto to achieve fairness and robustness simultaneously on a class of linear problems. Empirically, across a suite of federated datasets, we show that Ditto not only achieves competitive performance relative to recent personalization methods, but also enables more accurate, robust, and fair models relative to state-of-the-art fair or robust baselines.

研究の動機と目的

• 統計的に不均一なフェデレーテッドネットワークにおける公平性と頑健性のトレードオフへ対処する。
• 全体モデルへ向けて個別化モデルを正則化する、シンプルでかつスケーラブルな個別化フレームワーク（Ditto）を提案する。
• 収束保証を提供し、Dittoが精度・頑健性・公平性の点で最先端のベースラインと同等かそれを上回ることを示す。
• 多様なデータセットと攻撃にわたって、個別化がどのようにフェデレーテッドラーニングの頑健性と公平性を本質的に向上させ得るかを分析する。

提案手法

• Dittoを2階層の目的関数として定式化する：各デバイス k について、F_k(v_k) + (lambda/2) ||v_k - w^*||^2を最小化する。ここで w^* は、統合目的 G(F_1(w), ..., F_K(w)) の最適なグローバルモデルである。
• 交互最適化アルゴリズム（Algorithm 1）を用い、サーバがグローバルモデル w^t を標準的なFL解法で更新し、各デバイスが局所更新を用いて min_v_k F_k(v_k) + (lambda/2)||v_k - w^t||^2 を近似的に解く。
• グローバル目的の解法として任意の G(·) ソルバー（例：FedAvg）を使用できるようにしてモジュール性を維持し、プライバシーと通信特性を保持する。
• 収束保証を提供：F_k が強凸かつ滑らかで、w^t が w^* に収束すれば、v_k^t は v_k^* に収束し、関連した収束速度を示す（定理1）。
• Dittoの公平性/頑健性のトレードオフを線形問題のクラスで分析し、最適な lambda* を導出して Pareto 利得を示す。

実験結果

リサーチクエスチョン

• RQ1 Can personalized federated learning via global-regularized multi-task learning improve accuracy, robustness, and fairness simultaneously in heterogeneous FL settings?
• RQ2 How does the Ditto regularization parameter lambda balance local personalization versus global consistency to enhance fairness and defend against training-time attacks?
• RQ3 What convergence properties does the Ditto solver exhibit under common FL practices (limited device participation, local updates)?
• RQ4 Do the fairness and robustness benefits of Ditto generalize beyond convex settings to non-convex models and real-world FL benchmarks?

主な発見

Dataset	Attack level	Global	Local	Fair (TERM, t=1)	Ditto
Fashion MNIST	A1 clean	0.911 (0.08)	0.876 (0.10)	0.909 (0.07)	0.943 (0.06)
Fashion MNIST	A1 20% adversaries	0.897 (0.08)	0.874 (0.10)	0.751 (0.12)	0.944 (0.07)
Fashion MNIST	A1 50% adversaries	0.855 (0.10)	0.876 (0.11)	0.637 (0.13)	0.937 (0.07)
Fashion MNIST	A1 80% adversaries	0.753 (0.13)	0.879 (0.10)	0.547 (0.11)	0.907 (0.10)
Fashion MNIST	A2 20% adversaries	0.900 (0.08)	0.874 (0.10)	0.731 (0.13)	0.938 (0.07)
Fashion MNIST	A2 50% adversaries	0.882 (0.09)	0.876 (0.11)	0.637 (0.14)	0.930 (0.08)
Fashion MNIST	A2 80% adversaries	0.857 (0.10)	0.879 (0.10)	0.653 (0.13)	0.913 (0.09)
Fashion MNIST	A3 10% adversaries	0.753 (0.10)	0.874 (0.10)	0.601 (0.12)	0.921 (0.09)
Fashion MNIST	A3 20% adversaries	0.551 (0.13)	0.876 (0.10)	0.131 (0.16)	0.902 (0.09)
Fashion MNIST	A3 50% adversaries	0.275 (0.12)	0.874 (0.10)	0.131 (0.16)	0.873 (0.11)
FEMNIST	A1 clean	0.804 (0.11)	0.628 (0.15)	0.809 (0.11)	0.834 (0.09)
FEMNIST	A1 20% adversaries	0.773 (0.11)	0.620 (0.14)	0.636 (0.15)	0.802 (0.10)
FEMNIST	A1 50% adversaries	0.727 (0.12)	0.627 (0.14)	0.562 (0.13)	0.762 (0.11)
FEMNIST	A1 80% adversaries	0.574 (0.15)	0.607 (0.14)	0.478 (0.12)	0.672 (0.13)
FEMNIST	A2 10% adversaries	0.774 (0.11)	0.620 (0.14)	0.440 (0.15)	0.801 (0.09)
FEMNIST	A2 15% adversaries	0.703 (0.14)	0.627 (0.14)	0.336 (0.12)	0.700 (0.15)
FEMNIST	A2 20% adversaries	0.636 (0.15)	0.607 (0.14)	0.353 (0.12)	0.675 (0.14)
FEMNIST	A3 10% adversaries	0.517 (0.14)	0.607 (0.14)	0.316 (0.12)	0.685 (0.15)
FEMNIST	A3 15% adversaries	0.487 (0.14)	0.620 (0.14)	0.299 (0.11)	0.650 (0.14)
FEMNIST	A3 20% adversaries	0.314 (0.13)	0.620 (0.14)	0.316 (0.12)	0.613 (0.13)

• Ditto achieves competitive or superior accuracy compared with personalization methods and outperforms several robust or fair baselines on diverse FL datasets.
• Across attacks and datasets, Ditto improves test accuracy by approximately 6% (absolute) over the strongest robust baseline on average.
• Ditto reduces variance of test accuracy across devices by about 10% on average, indicating improved fairness.
• Ditto provides robustness without sacrificing much fairness, and in some cases improves both fairness and robustness over state-of-the-art baselines.
• A theoretical analysis on linear problems identifies an optimal lambda* that yields the most accurate, fair, and robust solution within Ditto’s space.