[Paper Review] Local Differential Privacy for Federated Learning

M.A.P. Chamikara, Dongxi Liu+5 more|arXiv (Cornell University)|Feb 12, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes LDPFL, a novel local differential privacy (LDP) framework for federated learning in cross-silo industrial settings. By applying randomized response and optimized unary encoding to a 1D flattened intermediate feature vector rather than model weights, LDPFL achieves high model utility (up to 98% accuracy) even under strict privacy budgets (e.g., ε = 0.5), outperforming existing LDP and GDP methods in privacy-utility trade-off.

ABSTRACT

Advanced adversarial attacks such as membership inference and model memorization can make federated learning (FL) vulnerable and potentially leak sensitive private data. Local differentially private (LDP) approaches are gaining more popularity due to stronger privacy notions and native support for data distribution compared to other differentially private (DP) solutions. However, DP approaches assume that the FL server (that aggregates the models) is honest (run the FL protocol honestly) or semi-honest (run the FL protocol honestly while also trying to learn as much information as possible). These assumptions make such approaches unrealistic and unreliable for real-world settings. Besides, in real-world industrial environments (e.g., healthcare), the distributed entities (e.g., hospitals) are already composed of locally running machine learning models (this setting is also referred to as the cross-silo setting). Existing approaches do not provide a scalable mechanism for privacy-preserving FL to be utilized under such settings, potentially with untrusted parties. This paper proposes a new local differentially private FL (named LDPFL) protocol for industrial settings. LDPFL can run in industrial settings with untrusted entities while enforcing stronger privacy guarantees than existing approaches. LDPFL shows high FL model performance (up to 98%) under small privacy budgets (e.g., epsilon = 0.5) in comparison to existing methods.

Motivation & Objective

• To address the lack of scalable, privacy-preserving FL mechanisms in industrial cross-silo settings with untrusted clients and servers.
• To overcome the limitations of existing LDP approaches in FL, including poor utility under small privacy budgets and inefficient handling of high-dimensional model parameters.
• To enable stronger privacy guarantees than global DP methods by eliminating reliance on a trusted curator or aggregator.
• To improve the privacy-utility trade-off in FL by applying LDP not to model weights but to intermediate feature representations.
• To provide a practical, efficient, and robust solution for real-world applications such as healthcare and smart agriculture where data is distributed and privacy is critical.

Proposed method

• LDPFL applies local differential privacy by randomizing a 1D flattened vector derived from the intermediate output of a locally trained deep neural network, rather than perturbing model parameters directly.
• It uses two LDP mechanisms—randomized response (RR) and optimized unary encoding (OUE)—to ensure differential privacy at the input level.
• The method trains a second fully connected DNN on the randomized 1D vector, enabling effective learning while preserving privacy.
• The architecture separates the client model into a CNN for feature extraction and a DNN for classification, with the randomization layer applied after feature flattening.
• Privacy budgets (ε) are applied efficiently by targeting the feature representation, reducing noise impact compared to weight-level perturbation.
• The approach supports untrusted clients and servers, as data never leaves the client’s device in raw form, and only randomized features are shared.

Experimental results

Research questions

• RQ1 Can a local differential privacy mechanism be applied effectively to intermediate feature representations in FL to improve utility under strict privacy budgets?
• RQ2 How does LDPFL compare to existing global and local DP methods in terms of model accuracy and privacy-utility trade-off?
• RQ3 Can LDPFL maintain high model performance (e.g., >95% accuracy) when ε is as low as 0.5, which is considered extremely strict in DP?
• RQ4 Does applying LDP at the feature level rather than the weight level lead to better privacy-utility balance in deep learning-based FL?
• RQ5 Can LDPFL be effectively deployed in untrusted cross-silo settings, such as healthcare or smart agriculture, without requiring a trusted server?

Key findings

• LDPFL achieves up to 98% test accuracy on CIFAR10 and SVHN datasets under a strict privacy budget of ε = 0.5, demonstrating high utility under strong privacy constraints.
• The method outperforms existing LDP-based FL approaches such as α-CLDP-Fed and DPSGD in both accuracy and privacy-utility balance, especially at low ε values.
• By randomizing intermediate features instead of model weights, LDPFL reduces the noise burden on high-dimensional parameter matrices, improving model convergence and performance.
• The use of optimized unary encoding (OUE) and randomized response (RR) enables effective privacy guarantees while preserving sufficient bit-level information for learning.
• Benchmarking shows LDPFL is preferred over SMC, GDP, and other LDP methods when high utility is required under strict privacy settings.
• The architecture is adaptable to various datasets, including MNIST, FMNIST, CIFAR10, and SVHN, with consistent performance across different input resolutions and model configurations.