[Paper Review] Byzantine-Robust Clustered Federated Learning

Zhixu Tao, Kun Yang+1 more|arXiv (Cornell University)|Jun 1, 2023

HIV, Drug Use, Sexual RiskMedicine3 citations

TL;DR

This paper proposes a Byzantine-robust iterative federated clustering algorithm that jointly identifies non-Byzantine machine clusters and optimizes local models despite adversarial attacks. By integrating coordinate-wise trimmed mean and median aggregation into the IFCA framework, it achieves convergence under strong convexity and outperforms existing methods, especially in high-dimensional or high-cluster settings with Byzantine faults.

ABSTRACT

This paper focuses on the problem of adversarial attacks from Byzantine machines in a Federated Learning setting where non-Byzantine machines can be partitioned into disjoint clusters. In this setting, non-Byzantine machines in the same cluster have the same underlying data distribution, and different clusters of non-Byzantine machines have different learning tasks. Byzantine machines can adversarially attack any cluster and disturb the training process on clusters they attack. In the presence of Byzantine machines, the goal of our work is to identify cluster membership of non-Byzantine machines and optimize the models learned by each cluster. We adopt the Iterative Federated Clustering Algorithm (IFCA) framework of Ghosh et al. (2020) to alternatively estimate cluster membership and optimize models. In order to make this framework robust against adversarial attacks from Byzantine machines, we use coordinate-wise trimmed mean and coordinate-wise median aggregation methods used by Yin et al. (2018). Specifically, we propose a new Byzantine-Robust Iterative Federated Clustering Algorithm to improve on the results in Ghosh et al. (2019). We prove a convergence rate for this algorithm for strongly convex loss functions. We compare our convergence rate with the convergence rate of an existing algorithm, and we demonstrate the performance of our algorithm on simulated data.

Motivation & Objective

• Address the challenge of Byzantine attacks in federated learning where a subset of machines can manipulate training by sending corrupted updates.
• Enable robust clustering of non-Byzantine machines into disjoint groups based on shared data distributions, even when Byzantine machines target and disrupt specific clusters.
• Design an iterative algorithm that jointly estimates cluster membership and optimizes local models under adversarial conditions.
• Achieve convergence for strongly convex loss functions without requiring equal data sizes across normal machines, only a minimum data point threshold.
• Improve upon prior work by reducing dependence on dimension and cluster size, especially in high-dimensional or large-cluster regimes.

Proposed method

• Adopt the Iterative Federated Clustering Algorithm (IFCA) framework to alternate between cluster assignment and model optimization.
• Integrate coordinate-wise trimmed mean and coordinate-wise median aggregation to mitigate the impact of Byzantine updates.
• Use loss function values at each iteration to assign non-Byzantine machines to clusters based on similarity to cluster centers.
• Allow Byzantine machines to compute gradients at 3× the cluster center to generate adversarial outliers, simulating worst-case attacks.
• Apply robust aggregation rules that discard the most extreme values in each coordinate across model updates to preserve convergence.
• Prove convergence under strong convexity with assumptions on the minimum number of data points per normal machine and worst-case Byzantine fraction.

Experimental results

Research questions

• RQ1 Can a federated clustering algorithm maintain convergence and accuracy when a fraction of clients are Byzantine and actively target specific clusters?
• RQ2 How does the performance of robust aggregation (trimmed mean/median) compare to standard FedAvg in the presence of Byzantine attacks?
• RQ3 What is the impact of high dimensionality and increasing number of clusters on the convergence and error rate of robust federated clustering?
• RQ4 Does the proposed algorithm achieve better error rates than existing methods like the Three-Stage Algorithm, particularly in high-dimensional or large-cluster settings?
• RQ5 What are the theoretical convergence guarantees of the proposed Byzantine-robust IFCA framework under strong convexity?

Key findings

• The proposed algorithm with coordinate-wise trimmed mean and median aggregation achieves convergence for strongly convex loss functions under Byzantine attacks.
• In low-dimensional settings (d ≤ 50), the Three-Stage Algorithm performs best, but its performance degrades significantly as dimension d increases due to suboptimal clustering in high dimensions.
• The IFCA framework using FedAvg performs worse than the proposed algorithm due to lack of robustness to Byzantine updates.
• For high-dimensional problems (d = 500) and increasing numbers of clusters (k = 2 to 15), the proposed algorithm consistently outperforms both IFCA with FedAvg and the Three-Stage Algorithm.
• The error metric dist = (1/k)∑‖θ̂j − θj*‖ shows that the proposed algorithm achieves lower average error across all tested settings, especially when d is large or k increases.
• The algorithm’s performance is robust to the fraction of Byzantine machines α = 0.05, and it maintains convergence even when all Byzantine machines collectively target the smallest cluster.