[Paper Review] Subsampled R\\'enyi Differential Privacy and Analytical Moments Accountant

Yu-Xiang Wang, Borja Balle+1 more|arXiv (Cornell University)|Jul 31, 2018

Privacy-Preserving Technologies in DataComputer Science95 citations

TL;DR

The paper derives a tight upper bound on Rényi differential privacy parameters for subsampled mechanisms in terms of the original mechanism’s RDP and the subsampling rate, and introduces an analytical moments accountant.

ABSTRACT

We study the problem of subsampling in differential privacy (DP), a question that is the centerpiece behind many successful differentially private machine learning algorithms. Specifically, we provide a tight upper bound on the R\\'enyi Differential Privacy (RDP) (Mironov, 2017) parameters for algorithms that: (1) subsample the dataset, and then (2) applies a randomized mechanism M to the subsample, in terms of the RDP parameters of M and the subsampling probability parameter. Our results generalize the moments accounting technique, developed by Abadi et al. (2016) for the Gaussian mechanism, to any subsampled RDP mechanism.

Motivation & Objective

• Motivate the need to understand privacy amplification under subsampling for Rényi DP (RDP).
• Derive a tight, general bound on the RDP parameters of subsampled mechanisms in terms of the original mechanism and sampling ratio.
• Propose an analytical moments accountant to track privacy parameters across compositions without predefined moment lists.
• Introduce a new ternary Pearson-Vajda divergence concept related to subsampling effects and connect it to RDP.
• Provide practical computational methods to output (epsilon, delta)-DP from RDP bounds and demonstrate improvements via experiments.

Proposed method

• Define subsampling without replacement and quantify privacy amplification for RDP (Theorem 9).
• Prove a tight upper bound on epsilon'(alpha) for M composed with subsample, in terms of epsilon(alpha), gamma, and epsilon(2) (and related terms).
• Show a lower bound (Proposition 11) indicating the bound is tight in general.
• Introduce a data structure for an analytical moments accountant that tracks the CGF K_M(lambda) symbolically and converts to (epsilon, delta)-DP efficiently.
• Discuss special cases including pure DP and Gaussian/Laplace mechanisms and outline asymptotic regimes for alpha and gamma.

Experimental results

Research questions

• RQ1 How does subsampling affect Rényi DP parameters for a general mechanism M?
• RQ2 Can we bound epsilon'(alpha) for M composed with subsample in terms of epsilon(alpha) and the subsampling ratio gamma?
• RQ3 Is the derived bound tight, and under what conditions can it be improved or matched by lower bounds?
• RQ4 How can we efficiently track privacy parameters across composition using an analytical moments accountant?
• RQ5 What are the practical implications and computational considerations for implementing these bounds in privacy-preserving ML pipelines?

Key findings

• A tight bound (Theorem 9) is provided for epsilon'(alpha) of M∘subsample in terms of gamma, alpha, and epsilon(·).
• The bound applies to any RDP mechanism, including Gaussian, Laplace, and exponential-family-based mechanisms.
• There is a phase transition in alpha for the amplification behavior: for small alpha the bound scales as O(alpha gamma^2), while for large alpha it can approach epsilon(alpha) or scale with gamma epsilon(∞).
• A lower bound (Proposition 11) demonstrates the upper bound cannot be improved in general without additional per-instance refinements.
• An analytical moments accountant is proposed, tracking CGFs symbolically to output (epsilon, delta)-DP efficiently without a fixed list of moments.
• The framework supports subsampling before applying M, enabling tighter privacy accounting in private learning settings and other DP applications.