[論文レビュー] Robust Federated Learning in a Heterogeneous Environment

Avishek Ghosh, Justin Hong+2 more|arXiv (Cornell University)|Jun 16, 2019

Machine Learning and Algorithms参考文献 38被引用数 150

ひとこと要約

本論文は、Byzantineワーカーを含む異種データ環境における頑健なフェデレーテッド学習のための three-stage モジュラー・フレームワークを提案し、局所モデルの頑健なクラスタリングと頑健な分散最適化を含む。これによりほぼ最適な誤差率を達成する。さらに頑健な Lloyd-type クラスタリングアルゴリズムを分析し、非頑健な手法に対して実用上顕著な利得を示す。

ABSTRACT

We study a recently proposed large-scale distributed learning paradigm, namely Federated Learning, where the worker machines are end users' own devices. Statistical and computational challenges arise in Federated Learning particularly in the presence of heterogeneous data distribution (i.e., data points on different devices belong to different distributions signifying different clusters) and Byzantine machines (i.e., machines that may behave abnormally, or even exhibit arbitrary and potentially adversarial behavior). To address the aforementioned challenges, first we propose a general statistical model for this problem which takes both the cluster structure of the users and the Byzantine machines into account. Then, leveraging the statistical model, we solve the robust heterogeneous Federated Learning problem \emph{optimally}; in particular our algorithm matches the lower bound on the estimation error in dimension and the number of data points. Furthermore, as a by-product, we prove statistical guarantees for an outlier-robust clustering algorithm, which can be considered as the Lloyd algorithm with robust estimation. Finally, we show via synthetic as well as real data experiments that the estimation error obtained by our proposed algorithm is significantly better than the non-Byzantine-robust algorithms; in particular, we gain at least by 53\% and 33\% for synthetic and real data experiments, respectively, in typical settings.

研究の動機と目的

• 実践的でスケーラブルな設定において、データ異質性とByzantine故障に対する頑健なフェデレーテッド学習を動機づける。
• local ERMs, robust clustering, and robust distributed optimizationを組み合わせた modular three-stage フレームワークを開発する。
• 次元とデータサンプルにおけるほぼ最適な推定誤差を示す理論的保証を提供する。
• 高次元に適した頑健な Lloyd ベースのクラスタリング手法を導入し分析する。
• 合成データと実データセットにおいて、非頑健なベースラインより経験的に改善を示す。

提案手法

• Stage I: 各ワーカーは局所経験的リスク最小化解 ERM ㅎerhatw(i) = argmin_w F(i)(w) where F(i)(w) = (1/n) sum_j f(w; x^{i,j}).
• Stage II: センターは ERMs を K クラスにクラスタリングする。外れ値耐性のある手法を用いる（n が大きい場合は閾値ベースのクラスタリング、あるいは頑健な Lloyd/K-means 変種）。
• Stage III: 各クラスタ内で頑健な分散最適化アルゴリズムを実行してクラスタ固有のモデルを得る（低〜中次元ではトリムド平均、高次元では反復フィルタリングなど）。
• 理論的結果は誤差境界を確立する: ||hatw_i - w_i*|| = O~( (alpha_hat_i * d)/sqrt(n) + d/sqrt(n M_i) ) under certain assumptions, with high probability.
• 高次元/外れ値設定に対しては、trimmed K-means を用いた頑健クラスタリングおよび反復フィルタリング手法への拡張分析を行う。
• クラスタリング分析には proper initialization によって誤クラスタリング点が指数的に減衰する頑健 Lloyd 系アルゴリズムを含む。

実験結果

リサーチクエスチョン

• RQ1 How can federated learning be made robust to Byzantine workers while handling heterogeneous (clustered) data distributions?
• RQ2 Can a modular three-stage framework (local ERMs, robust clustering, robust distributed optimization) achieve near-optimal estimation error under Byzantine and heterogeneity?
• RQ3 What are the theoretical guarantees for robust clustering of local models in the presence of adversaries, and how do they scale with dimension?
• RQ4 How do robust aggregation and clustering choices impact practical performance on synthetic and real datasets?
• RQ5 What are the performance gains over non-robust federated learning methods in heterogeneous environments?

主な発見

• The proposed three-stage algorithm achieves near-optimal error rates: ||hatw_i - w_i*|| = O~( (alpha_hat_i * d)/sqrt(n) + d/sqrt(n M_i) ).
• Robust clustering (trimmed K-means / K-geomedians) reduces misclustering compared to standard K-means, with exponential decay in misclustered points under proper initialization.
• Empirical results show significant improvements over non-robust baselines: estimation error reductions of at least 53% on synthetic data and 33% on Yahoo! Learning to Rank real data experiments.
• A robust Lloyd-style clustering analysis provides theoretical guarantees in high-dimensional settings, making the approach practical for large d.
• The framework remains modular, allowing different robust subroutines in each stage while maintaining overall performance guarantees.
• The Oracle-like comparison indicates the method matches the performance of an oracle that knows non-Byzantine cluster identities in many regimes.