[Paper Review] Federated Learning with Local Differential Privacy: Trade-offs between Privacy, Utility, and Communication

Muah Kim, Onur Günlü+1 more|arXiv (Cornell University)|Feb 9, 2021

Privacy-Preserving Technologies in Data19 references4 citations

TL;DR

This paper proposes a federated learning framework with local differential privacy (LDP) using Gaussian noise to balance privacy, utility, and communication efficiency. By employing an enhanced moments accountant for tighter privacy accounting and modeling query sensitivity as a variable, the method achieves significantly improved utility and reduced transmission rate compared to prior approaches, especially under strong privacy constraints.

ABSTRACT

Federated learning (FL) allows to train a massive amount of data privately due to its decentralized structure. Stochastic gradient descent (SGD) is commonly used for FL due to its good empirical performance, but sensitive user information can still be inferred from weight updates shared during FL iterations. We consider Gaussian mechanisms to preserve local differential privacy (LDP) of user data in the FL model with SGD. The trade-offs between user privacy, global utility, and transmission rate are proved by defining appropriate metrics for FL with LDP. Compared to existing results, the query sensitivity used in LDP is defined as a variable and a tighter privacy accounting method is applied. The proposed utility bound allows heterogeneous parameters over all users. Our bounds characterize how much utility decreases and transmission rate increases if a stronger privacy regime is targeted. Furthermore, given a target privacy level, our results guarantee a significantly larger utility and a smaller transmission rate as compared to existing privacy accounting methods.

Motivation & Objective

• To address the trade-off between privacy, utility, and communication cost in federated learning under local differential privacy (LDP).
• To develop a tighter privacy accounting method for stochastic gradient descent (SGD) in FL by leveraging the moments accountant with improved conversion from Rényi-DP to DP.
• To introduce a generic utility metric that accounts for heterogeneous user parameters such as dataset size, sampling probability, and individual privacy targets.
• To minimize transmission rate by modeling the differential entropy of noisy gradients under lossless communication.
• To demonstrate that the proposed method outperforms existing LDP-FL approaches in utility and communication efficiency under equivalent privacy budgets.

Proposed method

• Applies Gaussian randomization to local gradients to ensure local differential privacy (LDP) in federated SGD.
• Uses an enhanced moments accountant (MA) with optimal Rényi-DP to DP conversion for tighter privacy accounting, reducing overestimation of privacy loss.
• Defines query sensitivity as a variable parameter across users, enabling heterogeneous modeling of data sampling and privacy requirements.
• Derives a lower bound on utility that accounts for user-specific dataset sizes, sampling probabilities, and privacy levels.
• Models transmission rate as the differential entropy of noisy gradients, bounded using covariance matrix determinant and Gaussian entropy properties.
• Employs Hadamard’s inequality and AM-GM inequality to bound the determinant of the gradient covariance matrix, enabling tractable entropy analysis.

Experimental results

Research questions

• RQ1 How does variable query sensitivity across users affect the utility-privacy-communication trade-off in LDP-FL?
• RQ2 Can tighter privacy accounting via the enhanced moments accountant improve utility and reduce communication cost in LDP-FL?
• RQ3 What is the theoretical lower bound on model utility when users have heterogeneous data sizes, sampling probabilities, and privacy targets?
• RQ4 How does the differential entropy of noisy gradients scale with noise variance and model dimension in LDP-FL?
• RQ5 To what extent does the proposed method reduce required noise power and transmission rate compared to existing LDP-FL baselines?

Key findings

• The proposed method achieves a significantly larger utility bound compared to existing privacy accounting methods, especially under strong privacy regimes.
• The transmission rate is reduced due to tighter bounds on the differential entropy of noisy gradients, enabled by improved covariance matrix and noise entropy modeling.
• The method guarantees a lower bound on convergence rate that scales as $ \mathcal{O}(1/t) $, with a constant factor depending on $ \max\{2, 1 + d\sigma^2\} $, improving over prior bounds.
• For heterogeneous users, the utility bound explicitly incorporates distinct dataset sizes, sampling probabilities, and privacy levels, enabling realistic system modeling.
• The enhanced moments accountant with optimal conversion reduces overestimation of privacy loss, allowing higher utility at the same privacy budget.
• Numerical comparisons show gains in required noise power, utility, and transmission rate, demonstrating the method's superiority over SC, AC1, and AC2-based approaches.