[Paper Review] Faithful Edge Federated Learning: Scalability and Privacy

Meng Zhang, Ermin Wei+1 more|arXiv (Cornell University)|Jun 30, 2021

Privacy-Preserving Technologies in Data48 references4 citations

TL;DR

This paper proposes Faithful Edge Federated Learning (FFL) and its differentially private variant (DP-FFL), the first mechanisms ensuring truthful participation and execution in federated learning under non-i.i.d. and unbalanced data. By approximating VCG payments via incremental computation and clustering, FFL achieves $Ø(\log K)$ time complexity, faithful implementation, voluntary participation, and privacy, with a three-way tradeoff among privacy, convergence speed, and payment accuracy loss.

ABSTRACT

Federated learning enables machine learning algorithms to be trained over a network of multiple decentralized edge devices without requiring the exchange of local datasets. Successfully deploying federated learning requires ensuring that agents (e.g., mobile devices) faithfully execute the intended algorithm, which has been largely overlooked in the literature. In this study, we first use risk bounds to analyze how the key feature of federated learning, unbalanced and non-i.i.d. data, affects agents' incentives to voluntarily participate and obediently follow traditional federated learning algorithms. To be more specific, our analysis reveals that agents with less typical data distributions and relatively more samples are more likely to opt out of or tamper with federated learning algorithms. To this end, we formulate the first faithful implementation problem of federated learning and design two faithful federated learning mechanisms which satisfy economic properties, scalability, and privacy. Further, the time complexity of computing all agents' payments in the number of agents is $\mathcal{O}(1)$. First, we design a Faithful Federated Learning (FFL) mechanism which approximates the Vickrey-Clarke-Groves (VCG) payments via an incremental computation. We show that it achieves (probably approximate) optimality, faithful implementation, voluntary participation, and some other economic properties (such as budget balance). Second, by partitioning agents into several subsets, we present a scalable VCG mechanism approximation. We further design a scalable and Differentially Private FFL (DP-FFL) mechanism, the first differentially private faithful mechanism, that maintains the economic properties. Our mechanism enables one to make three-way performance tradeoffs among privacy, the iterations needed, and payment accuracy loss.

Motivation & Objective

• To address the lack of incentive mechanisms ensuring agents faithfully execute federated learning despite non-i.i.d. and unbalanced data distributions.
• To design a mechanism that guarantees voluntary participation and faithful implementation under strategic agent behavior.
• To achieve scalability and differential privacy while preserving key economic properties such as budget balance and approximate optimality.
• To enable a three-way tradeoff among privacy, convergence speed, and payment accuracy loss in a differentially private setting.

Proposed method

• Proposes a Faithful Federated Learning (FFL) mechanism that approximates Vickrey–Clarke–Groves (VCG) payments through incremental computation to ensure truthful reporting and faithful execution.
• Introduces a clustered VCG approximation to scale the mechanism, reducing time complexity to $Ø(\log K)$ in the number of agents $K$.
• Designs a differentially private variant, DP-FFL, by adding calibrated noise to payments while preserving economic properties and enabling privacy-accuracy-convergence tradeoffs.
• Employs risk bounds to analyze agents’ incentives under non-i.i.d. data, revealing that agents with less typical data and more samples are more likely to opt out or manipulate.
• Uses a payment mechanism based on agents’ marginal contributions to global model improvement, computed via incremental updates to reduce computational cost.
• Applies differential privacy to the payment mechanism by adding Laplace noise, ensuring individual agent contributions remain hidden.

Experimental results

Research questions

• RQ1 How do unbalanced and non-i.i.d. data distributions disincentivize agents from voluntarily participating or faithfully executing federated learning algorithms?
• RQ2 Can a mechanism be designed to ensure faithful implementation while maintaining scalability and privacy in edge federated learning?
• RQ3 What tradeoffs exist between privacy, convergence speed, and payment accuracy loss in a differentially private faithful federated learning mechanism?
• RQ4 How does the proposed mechanism compare to baseline federated learning (e.g., FedAvg) and local learning in terms of performance and agent incentives?

Key findings

• The FFL mechanism achieves (probably approximate) optimality, faithful implementation, voluntary participation, and budget balance, with time complexity of $Ø(\log K)$ in the number of agents.
• The DP-FFL mechanism is the first differentially private faithful federated learning mechanism, enabling a three-way tradeoff among privacy, convergence iterations, and payment accuracy loss.
• Both FFL and DP-FFL significantly outperform manipulated FedAvg and local learning in terms of global loss and test accuracy, especially under moderate data heterogeneity ($\Delta \leq 0.15$).
• When data heterogeneity is high ($\Delta \geq 0.15$), local learning may outperform federated learning, but FFL and DP-FFL still provide better outcomes than manipulated FedAvg.
• Payment accuracy loss decreases with higher privacy risk $\alpha$ and with more clusters $L$, with $L \geq 4$ sufficient to maintain low loss.
• The number of training samples $n$ has a greater impact on local learning and manipulated FedAvg than on FFL or DP-FFL, indicating the proposed mechanisms are more robust to data imbalance.