[Paper Review] Can You Really Backdoor Federated Learning?

Ziteng Sun, Peter Kairouz+2 more|arXiv (Cornell University)|Nov 18, 2019

Privacy-Preserving Technologies in Data31 references368 citations

TL;DR

This paper studies backdoor (targeted) model-update poisoning attacks in federated learning, analyzes attack models under EMNIST with realistic non-iid data, and evaluates defenses like norm clipping and weak differential privacy.

ABSTRACT

The decentralized nature of federated learning makes detecting and defending against adversarial attacks a challenging task. This paper focuses on backdoor attacks in the federated learning setting, where the goal of the adversary is to reduce the performance of the model on targeted tasks while maintaining good performance on the main task. Unlike existing works, we allow non-malicious clients to have correctly labeled samples from the targeted tasks. We conduct a comprehensive study of backdoor attacks and defenses for the EMNIST dataset, a real-life, user-partitioned, and non-iid dataset. We observe that in the absence of defenses, the performance of the attack largely depends on the fraction of adversaries present and the "complexity'' of the targeted task. Moreover, we show that norm clipping and "weak'' differential privacy mitigate the attacks without hurting the overall performance. We have implemented the attacks and defenses in TensorFlow Federated (TFF), a TensorFlow framework for federated learning. In open-sourcing our code, our goal is to encourage researchers to contribute new attacks and defenses and evaluate them on standard federated datasets.

Motivation & Objective

• Motivate and formalize backdoor attacks in federated learning with non-malicious targeted-task samples.
• Quantify how attack success depends on adversary fraction and targeted task complexity.
• Evaluate defenses such as norm clipping and weak differential privacy on realistic federated settings.

Proposed method

• Model update poisoning attacks that replace the model with a backdoored version using a boosted update from adversaries.
• Attack models include random sampling of compromised clients and fixed-frequency attackers.
• Backdoor task defined by mislabeling a targeted class (e.g., 7s as 1s) across multiple target clients.
• Defense mechanisms include norm-based update clipping and adding Gaussian noise (weak DP).
• Experiments conducted on EMNIST with a 5-layer CNN in TensorFlow Federated, varying backdoor task count and attacker fraction.

Experimental results

Research questions

• RQ1 How effective are backdoor (targeted) model update poisoning attacks in federated learning under realistic EMNIST distributions?
• RQ2 How do attacker fraction and backdoor task complexity affect attack success rates?
• RQ3 Can norm clipping and weak differential privacy mitigate backdoor attacks without severely hurting main task performance?
• RQ4 What is the impact of the number of backdoor tasks on the ability to backdoor a model under federated learning?
• RQ5 How do random sampling and fixed-frequency attacker models compare in effectiveness?

Key findings

• Backdoor attack success depends largely on the fraction of adversaries in the system.
• Attacks remain effective only when a non-negligible fraction of clients are compromised (e.g., below 1% reduces effectiveness).
• Norm clipping substantially reduces backdoor success by bounding update norms.
• Adding a small amount of Gaussian noise (weak DP) further mitigates attacks with limited impact on main task performance.
• Increasing the number of backdoor tasks makes it harder to fit a malicious model while maintaining main task accuracy.
• Fixed-frequency attacks are slightly more effective than random sampling in their experiments.