[Paper Review] Source Inference Attacks in Federated Learning

Hongsheng Hu, Zoran Salčić+3 more|arXiv (Cornell University)|Sep 13, 2021

Privacy-Preserving Technologies in Data38 references4 citations

TL;DR

This paper proposes Source Inference Attacks (SIAs) in federated learning, where an honest-but-curious server infers which client contributed a given training example by analyzing local model prediction losses. The attack achieves high accuracy—up to 75.1% ASR—without violating FL protocols, revealing that source privacy is a critical yet overlooked threat beyond membership inference.

ABSTRACT

Federated learning (FL) has emerged as a promising privacy-aware paradigm that allows multiple clients to jointly train a model without sharing their private data. Recently, many studies have shown that FL is vulnerable to membership inference attacks (MIAs) that can distinguish the training members of the given model from the non-members. However, existing MIAs ignore the source of a training member, i.e., the information of which client owns the training member, while it is essential to explore source privacy in FL beyond membership privacy of examples from all clients. The leakage of source information can lead to severe privacy issues. For example, identification of the hospital contributing to the training of an FL model for COVID-19 pandemic can render the owner of a data record from this hospital more prone to discrimination if the hospital is in a high risk region. In this paper, we propose a new inference attack called source inference attack (SIA), which can derive an optimal estimation of the source of a training member. Specifically, we innovatively adopt the Bayesian perspective to demonstrate that an honest-but-curious server can launch an SIA to steal non-trivial source information of the training members without violating the FL protocol. The server leverages the prediction loss of local models on the training members to achieve the attack effectively and non-intrusively. We conduct extensive experiments on one synthetic and five real datasets to evaluate the key factors in an SIA, and the results show the efficacy of the proposed source inference attack.

Motivation & Objective

• To address the gap in federated learning privacy by investigating source privacy—i.e., identifying which client contributed a training example—beyond existing membership inference attacks.
• To demonstrate that a server can infer source information without modifying the FL protocol or degrading model performance.
• To evaluate the effectiveness of source inference under diverse FL settings, including non-i.i.d. data distributions and varying client numbers.
• To explore the limitations of differential privacy as a defense, revealing a significant trade-off between model utility and attack mitigation.

Proposed method

• The attack is formulated from a Bayesian perspective, modeling the posterior probability of a data sample's source based on local model prediction losses.
• The server computes the prediction loss of each local model on a given training example and uses this to estimate the most likely client that contributed it.
• The method is non-intrusive and does not alter model updates or communication—only leveraging existing FL communication patterns.
• The attack is evaluated across one synthetic and five real-world datasets, including Location and CIFAR-10, under varying data skew (α), number of clients (K), and local epochs (E).
• Differential privacy is applied as a defense mechanism to test its efficacy, with hyperparameters tuned to balance privacy and model utility.

Experimental results

Research questions

• RQ1 Can a server in federated learning infer which client contributed a specific training example without violating the protocol?
• RQ2 How does data non-i.i.d. distribution and model generalization affect the success of source inference attacks?
• RQ3 What is the impact of the number of clients, local training epochs, and data skew on source inference accuracy?
• RQ4 Can differential privacy effectively defend against source inference attacks without severely degrading model utility?

Key findings

• The source inference attack achieves an average success rate (ASR) of 75.1% on the Location dataset and 55.2% on CIFAR-10 under standard FL settings.
• The success of the attack is strongly correlated with the generalizability of local models and the diversity of local data distributions.
• When data is highly non-i.i.d. (low α), the attack performance increases, indicating that data skew enhances source leakage.
• Differential privacy reduces the ASR to 55.1% on Location and 24.4% on CIFAR-10, but at the cost of dropping test accuracy from 71.2% to 12.3% and 68.3% to 10.0%, respectively.
• The results show that vanilla differential privacy is ineffective as a defense for SIAs due to the severe utility degradation, indicating a need for better privacy-preserving mechanisms.