[Paper Review] Hierarchical Federated Learning with Privacy

Varun Chandrasekaran, Suman Banerjee+2 more|arXiv (Cornell University)|Jun 10, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper introduces hierarchical federated learning with differential privacy (HDP), where clients form zones aggregated by intermediate super-nodes before reaching a central aggregator. By adding calibrated noise at the super-node level, HDP achieves better privacy-utility trade-offs than local or central DP, matching central DP's model accuracy while providing stronger resilience against inference attacks than local DP.

ABSTRACT

Federated learning (FL), where data remains at the federated clients, and where only gradient updates are shared with a central aggregator, was assumed to be private. Recent work demonstrates that adversaries with gradient-level access can mount successful inference and reconstruction attacks. In such settings, differentially private (DP) learning is known to provide resilience. However, approaches used in the status quo (\ie central and local DP) introduce disparate utility vs. privacy trade-offs. In this work, we take the first step towards mitigating such trade-offs through {\em hierarchical FL (HFL)}. We demonstrate that by the introduction of a new intermediary level where calibrated DP noise can be added, better privacy vs. utility trade-offs can be obtained; we term this {\em hierarchical DP (HDP)}. Our experiments with 3 different datasets (commonly used as benchmarks for FL) suggest that HDP produces models as accurate as those obtained using central DP, where noise is added at a central aggregator. Such an approach also provides comparable benefit against inference adversaries as in the local DP case, where noise is added at the federated clients.

Motivation & Objective

• To address the inherent privacy-utility trade-off in federated learning (FL) under differential privacy (DP), especially when central or local DP mechanisms lead to suboptimal model utility.
• To explore whether introducing a hierarchical structure—where clients are grouped into zones and aggregated by super-nodes—can improve privacy-utility trade-offs while maintaining formal DP guarantees.
• To formalize a new mechanism, hierarchical DP (HDP), where noise is injected at the super-node level to amplify privacy and improve model utility compared to existing DP approaches in FL.
• To evaluate the resilience of HDP against reconstruction and inference attacks by adversaries at different trust levels (e.g., super-node or central aggregator)
• To analyze the impact of system parameters such as client sampling rate (k) and zone structure on privacy and model utility

Proposed method

• Proposes a hierarchical FL architecture where clients are grouped into zones, each managed by a super-node that aggregates client gradients before forwarding to the central aggregator.
• Introduces hierarchical differential privacy (HDP), where calibrated Gaussian noise is added at the super-node level to ensure DP guarantees at the zone level.
• Leverages the composability of the Gaussian mechanism to amplify privacy: noise at the super-node level reduces the effective privacy loss at the central aggregator due to the aggregation of already noisy gradients.
• Employs secure aggregation at the super-node level to prevent individual client gradients from being exposed, reducing attack surface for reconstruction adversaries.
• Adapts standard DP accounting to the hierarchical setting, enabling formal privacy budget (ε) tracking across multiple levels of aggregation.
• Uses state-of-the-art implementations for super-node election and secure aggregation to ensure practicality and low runtime overhead

Experimental results

Research questions

• RQ1 Can introducing an intermediate super-node layer in federated learning improve the privacy-utility trade-off compared to central or local DP?
• RQ2 How does hierarchical DP (HDP) compare to local DP and central DP in terms of model accuracy and resilience against inference and reconstruction attacks?
• RQ3 What is the impact of zone structure and client sampling rate (k) on the final model’s privacy and utility in HDP?
• RQ4 Can privacy amplification be achieved through the hierarchical aggregation process, particularly via the composability of the Gaussian mechanism?
• RQ5 How do different threat models (e.g., adversaries at super-node or central aggregator level) affect information leakage in HDP, and can HDP defend against them?

Key findings

• HDP achieves model accuracy comparable to central DP (CDP), where noise is added at the central aggregator, while offering stronger privacy guarantees than local DP (LDP).
• HDP provides comparable defense against inference adversaries as local DP, but with significantly better model utility, making it a preferred alternative when feasible.
• The hierarchical structure enables privacy amplification: when the number of zones (s) is less than the total number of clients (k), the central aggregator receives only aggregated, noisy gradients, reducing the effective privacy loss.
• Experiments on CIFAR-10, CIFAR-100, and EMNIST datasets show that increasing the number of online clients (k) has minimal impact on model accuracy, suggesting robustness to client sampling fluctuations.
• HDP effectively thwarts reconstruction attacks by adversaries at the super-node level, even when a malicious client is elected as a super-node, due to secure aggregation and noise injection.
• The scheme remains robust under various threat models, including passive adversaries aiming for data reconstruction, and shows reduced information leakage when super-nodes are elected randomly and rotated over time.