[Paper Review] The Skellam Mechanism for Differentially Private Federated Learning

Naman Agarwal, Peter Kairouz+1 more|arXiv (Cornell University)|Oct 11, 2021

Privacy-Preserving Technologies in DataComputer Science43 references37 citations

TL;DR

Introduces the multi-dimensional Skellam mechanism for discrete differential privacy, analyzes its Rényi DP guarantees, and demonstrates competitive privacy-utility performance in federated learning with secure aggregation under communication constraints.

ABSTRACT

We introduce the multi-dimensional Skellam mechanism, a discrete differential privacy mechanism based on the difference of two independent Poisson random variables. To quantify its privacy guarantees, we analyze the privacy loss distribution via a numerical evaluation and provide a sharp bound on the Rényi divergence between two shifted Skellam distributions. While useful in both centralized and distributed privacy applications, we investigate how it can be applied in the context of federated learning with secure aggregation under communication constraints. Our theoretical findings and extensive experimental evaluations demonstrate that the Skellam mechanism provides the same privacy-accuracy trade-offs as the continuous Gaussian mechanism, even when the precision is low. More importantly, Skellam is closed under summation and sampling from it only requires sampling from a Poisson distribution -- an efficient routine that ships with all machine learning and data analysis software packages. These features, along with its discrete nature and competitive privacy-accuracy trade-offs, make it an attractive practical alternative to the newly introduced discrete Gaussian mechanism.

Motivation & Objective

• Motivate the need for a discrete, easily samplable DP mechanism suitable for high-dimensional federated learning and secure aggregation.
• Propose the multi-dimensional Skellam mechanism based on the difference of two Poisson variables as a noise model.
• Provide tight privacy guarantees via Rényi DP and privacy loss distributions, aiming to match Gaussian trade-offs.
• Demonstrate practical applicability in centralized and distributed DP settings, including communication-constrained FL scenarios.

Proposed method

• Define the multidimensional Skellam distribution and show closure under sum to enable distributed analysis.
• Develop the Skellam Mechanism by adding Skellam noise to integer-valued queries.
• Characterize privacy via privacy loss distributions and prove a sharp (α, ε)-RDP bound for integer α>1.
• Show that the Skellam mechanism’s RDP guarantees are within a 1+O(1/μ) factor of the Gaussian mechanism.
• Apply modular arithmetic and discretization to adapt Skellam for federated learning with secure aggregation.
• Compare Skellam to discrete Gaussian and binomial mechanisms to argue practical advantages.
• Provide end-to-end FL algorithm incorporating discretization, rounding, and Skellam noise under communication constraints.

Experimental results

Research questions

• RQ1 Can a discrete, closed-under-summation noise mechanism achieve similar privacy-utility trade-offs as Gaussian noise in high-dimensional, iterative learning?
• RQ2 How can Skellam noise be quantified and composed effectively using Rényi DP in multi-dimensional settings?
• RQ3 Is it feasible to deploy Skellam-based DP in federated learning with secure aggregation under communication constraints without sacrificing utility?
• RQ4 How does discretization, scaling, and rounding impact sensitivity and privacy in practical FL pipelines?
• RQ5 How does Skellam compare to discrete Gaussian and binomial mechanisms in terms of sampling, summation, and software support?
• RQ6

Key findings

• The multi-dimensional Skellam mechanism provides (α, ε(α))-RDP with ε(α) bounded close to the Gaussian baseline, up to 1+O(1/μ).
• Skellam noise is closed under summation, enabling efficient distributed DP with secure aggregation.
• Discrete Skellam sampling reduces to Poisson sampling, aligning with readily available software and hardware support.
• The mechanism matches Gaussian privacy-accuracy trade-offs in federated learning under tight privacy and communication budgets.
• Discretization and rounding strategies can bound sensitivity and maintain utility in central and distributed DP settings.
• Empirical evaluations on distributed mean estimation and federated datasets (Federated EMNIST, Stack Overflow, Shakespeare) show competitive performance with Gaussian baselines.