[论文解读] Subsampled R\\'enyi Differential Privacy and Analytical Moments Accountant

Yu-Xiang Wang, Borja Balle+1 more|arXiv (Cornell University)|Jul 31, 2018

Privacy-Preserving Technologies in Data被引用 95

一句话总结

该论文给出对子采样机制的 Rényi 差分隐私参数的紧上界，使用原始机制的 RDP 和子采样率，并引入一种分析性矩账户。

ABSTRACT

We study the problem of subsampling in differential privacy (DP), a question that is the centerpiece behind many successful differentially private machine learning algorithms. Specifically, we provide a tight upper bound on the R\\'enyi Differential Privacy (RDP) (Mironov, 2017) parameters for algorithms that: (1) subsample the dataset, and then (2) applies a randomized mechanism M to the subsample, in terms of the RDP parameters of M and the subsampling probability parameter. Our results generalize the moments accounting technique, developed by Abadi et al. (2016) for the Gaussian mechanism, to any subsampled RDP mechanism.

研究动机与目标

• Motivate the need to understand privacy amplification under subsampling for Rényi DP (RDP).
• Derive a tight, general bound on the RDP parameters of subsampled mechanisms in terms of the original mechanism and sampling ratio.
• Propose an analytical moments accountant to track privacy parameters across compositions without predefined moment lists.
• Introduce a new ternary Pearson-Vajda divergence concept related to subsampling effects and connect it to RDP.
• Provide practical computational methods to output (epsilon, delta)-DP from RDP bounds and demonstrate improvements via experiments.

提出的方法

• Define subsampling without replacement and quantify privacy amplification for RDP (Theorem 9).
• Prove a tight upper bound on epsilon'(alpha) for M composed with subsample, in terms of epsilon(alpha), gamma, and epsilon(2) (and related terms).
• Show a lower bound (Proposition 11) indicating the bound is tight in general.
• Introduce a data structure for an analytical moments accountant that tracks the CGF K_M(lambda) symbolically and converts to (epsilon, delta)-DP efficiently.
• Discuss special cases including pure DP and Gaussian/Laplace mechanisms and outline asymptotic regimes for alpha and gamma.

实验结果

研究问题

• RQ1 How does subsampling affect Rényi DP parameters for a general mechanism M?
• RQ2 Can we bound epsilon'(alpha) for M composed with subsample in terms of epsilon(alpha) and the subsampling ratio gamma?
• RQ3 Is the derived bound tight, and under what conditions can it be improved or matched by lower bounds?
• RQ4 How can we efficiently track privacy parameters across composition using an analytical moments accountant?
• RQ5 What are the practical implications and computational considerations for implementing these bounds in privacy-preserving ML pipelines?

主要发现

• A tight bound (Theorem 9) is provided for epsilon'(alpha) of M∘subsample in terms of gamma, alpha, and epsilon(·).
• The bound applies to any RDP mechanism, including Gaussian, Laplace, and exponential-family-based mechanisms.
• There is a phase transition in alpha for the amplification behavior: for small alpha the bound scales as O(alpha gamma^2), while for large alpha it can approach epsilon(alpha) or scale with gamma epsilon(∞).
• A lower bound (Proposition 11) demonstrates the upper bound cannot be improved in general without additional per-instance refinements.
• An analytical moments accountant is proposed, tracking CGFs symbolically to output (epsilon, delta)-DP efficiently without a fixed list of moments.
• The framework supports subsampling before applying M, enabling tighter privacy accounting in private learning settings and other DP applications.