[Paper Review] Rényi Differential Privacy of the Sampled Gaussian Mechanism

Ilya Mironov, Kunal Talwar+1 more|arXiv (Cornell University)|Aug 28, 2019

Privacy-Preserving Technologies in Data10 references88 citations

TL;DR

The paper analyzes the Rényi differential privacy (RDP) of the Sampled Gaussian Mechanism (SGM), unifying prior results, providing a nearly tight closed-form bound, and offering a numerically stable method to compute SGM’s RDP.

ABSTRACT

The Sampled Gaussian Mechanism (SGM)---a composition of subsampling and the additive Gaussian noise---has been successfully used in a number of machine learning applications. The mechanism's unexpected power is derived from privacy amplification by sampling where the privacy cost of a single evaluation diminishes quadratically, rather than linearly, with the sampling rate. Characterizing the precise privacy properties of SGM motivated development of several relaxations of the notion of differential privacy. This work unifies and fills in gaps in published results on SGM. We describe a numerically stable procedure for precise computation of SGM's Rényi Differential Privacy and prove a nearly tight (within a small constant factor) closed-form bound.

Motivation & Objective

• Motivate and analyze privacy of the Sampled Gaussian Mechanism (SGM), which combines subsampling with Gaussian noise.
• Unify existing results on SGM under Rényi differential privacy (RDP).
• Provide both closed-form bounds and numerically stable methods to compute SGM’s RDP.
• Clarify relationships among DP relaxations and privacy accounting approaches in the SGM context.

Proposed method

• Reduce the RDP analysis of SGM to a simple mixture of one-dimensional Gaussians and apply quasi-convexity of Rényi divergence to bound the divergences.
• Derive a closed-form bound by bounding Aα and Bα, where Aα = E_{z~N(0,σ^2)}[(μ(z)/μ0(z))^α] and Bα = E_{z~μ}[(μ0(z)/μ(z))^α] with μ0=,N(0,σ^2) and μ=(1−q)μ0+qμ1, μ1=N(1,σ^2).
• Prove that Aα ≥ Bα and use this to simplify the analysis.
• Provide a numerically stable computation method for Aα using either integer α (via binomial expansion) or a convergent series for fractional α and show how to evaluate it in practice.
• State a main theorem giving an RDP bound under parameter regimes (q ≤ 1/5, σ ≥ 4, and α within specified range) with ε = 2q^2α/σ^2.
• Discuss connections between different privacy notions (CDP, zCDP, tCDP, RDP) and the privacy accountant framework.

Experimental results

Research questions

• RQ1 What are tight or near-tight Rényi differential privacy bounds for the Sampled Gaussian Mechanism under various sampling rates q and noise levels σ?
• RQ2 How can the analysis be reduced to simple one-dimensional Gaussian mixtures to facilitate computation of RDP?
• RQ3 How do Aα and Bα compare, and how does their relationship affect the RDP of SGM?
• RQ4 Can we derive a numerically stable procedure to compute Aα exactly for both integer and non-integer α?
• RQ5 How do RDP results relate to and contrast with other DP relaxations and accounting methods in the context of SGM?

Key findings

• SGM with ℓ2-sensitivity 1 satisfies (α, ε)-RDP if ε ≤ (1/(α−1)) log max(Aα, Bα) and Aα ≥ Bα.
• A closed-form bound is obtained for Aα leading to an (α, ε)-RDP guarantee with ε = 2q^2α/σ^2 under specified parameter constraints (q ≤ 1/5, σ ≥ 4, and α within a defined range).
• A numerically stable procedure is provided to compute Aα exactly, using either a binomial expansion for integer α or a convergent series for fractional α, enabling precise RDP accounting.
• A general theorem shows that reducing to a simple one-dimensional Gaussian mixture captures the worst-case RDP bound for SGM under the stated assumptions.
• The work clarifies the relationships among CDP, zCDP, tCDP, and RDP and discusses how the moments accountant can convert RDP bounds to (ε, δ)-DP.