[Paper Review] Differentially Private Learning Needs Hidden State (Or Much Faster Convergence)

Jiayuan Ye, Reza Shokri|arXiv (Cornell University)|Mar 10, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper proposes a novel privacy analysis for differentially private stochastic gradient descent (DP-SGD) that leverages hidden model states and privacy amplification via sub-sampling and randomized post-processing. By modeling the privacy dynamics under 'shuffle and partition' and 'sample without replacement' mini-batch schemes, the authors prove that privacy bounds converge exponentially fast for strongly convex smooth losses, significantly improving over composition-based bounds—especially after a few training epochs.

ABSTRACT

Prior work on differential privacy analysis of randomized SGD algorithms relies on composition theorems, where the implicit (unrealistic) assumption is that the internal state of the iterative algorithm is revealed to the adversary. As a result, the Rényi DP bounds derived by such composition-based analyses linearly grow with the number of training epochs. When the internal state of the algorithm is hidden, we prove a converging privacy bound for noisy stochastic gradient descent (on strongly convex smooth loss functions). We show how to take advantage of privacy amplification by sub-sampling and randomized post-processing, and prove the dynamics of privacy bound for "shuffle and partition" and "sample without replacement" stochastic mini-batch gradient descent schemes. We prove that, in these settings, our privacy bound converges exponentially fast and is substantially smaller than the composition bounds, notably after a few number of training epochs. Thus, unless the DP algorithm converges fast, our privacy analysis shows that hidden state analysis can significantly amplify differential privacy.

Motivation & Objective

• Address the limitation of composition-based privacy analysis in DP-SGD, which overestimates privacy loss due to assuming full access to internal algorithm states.
• Improve the privacy-accuracy trade-off in differentially private learning by analyzing only the final model parameters (hidden state), which better reflects practical deployment.
• Develop tighter privacy bounds for multi-epoch DP-SGD by exploiting privacy amplification from sub-sampling and randomized post-processing.
• Provide a rigorous analysis for both 'shuffle and partition' and 'sample without replacement' mini-batch training schemes under hidden-state assumptions.
• Demonstrate that hidden-state analysis yields substantially smaller privacy bounds than composition theorems, especially when convergence is slow.

Proposed method

• Model noisy stochastic gradient descent under the assumption that only the final model parameters are released, hiding the internal state of the training process.
• Introduce a new privacy amplification bound for randomized post-processing by decomposing a noisy GD update into two steps: a smaller-noise update followed by pure Gaussian noise.
• Prove new Rényi differential privacy (RDP) bounds using sub-sampling and post-processing amplification for 'shuffle and partition' and 'sample without replacement' mini-batch selection.
• Apply the analysis to strongly convex and smooth loss functions, including logistic regression with clipped gradients to ensure finite sensitivity.
• Derive a tighter bound than prior work [15] by using a novel decomposition technique that separates noise scaling from gradient updates.
• Validate the theoretical bounds through analysis of gradient sensitivity and Hessian properties in logistic regression, ensuring smoothness and strong convexity.

Experimental results

Research questions

• RQ1 Can hidden-state privacy analysis yield significantly tighter privacy bounds than composition-based methods in multi-epoch DP-SGD?
• RQ2 How does privacy amplification via sub-sampling and randomized post-processing affect the convergence of privacy bounds in stochastic gradient descent?
• RQ3 What is the impact of mini-batch selection strategies—'shuffle and partition' and 'sample without replacement'—on the dynamics of privacy loss in DP-SGD?
• RQ4 Can the privacy bound converge exponentially fast under hidden-state assumptions, especially for slowly converging models?
• RQ5 How does the proposed method improve upon prior work [15] in terms of tightness and generality for stochastic mini-batch training?

Key findings

• The proposed privacy bound converges exponentially fast under 'shuffle and partition' and 'sample without replacement' schemes, unlike composition-based bounds that grow linearly with epochs.
• For strongly convex smooth loss functions, the privacy bound is substantially smaller than composition-based bounds after only a few training epochs.
• The new analysis achieves a tighter bound than prior work [15] for full gradient descent by using a novel decomposition of noisy updates into smaller-noise and post-processing steps.
• The method enables significantly improved privacy-accuracy trade-offs in DP-SGD, particularly for models with slow to moderate convergence.
• The privacy amplification by sub-sampling and post-processing is effectively leveraged in the hidden-state setting, leading to exponential decay in privacy loss over time.
• The theoretical framework applies to practical settings such as logistic regression with clipped gradients, ensuring finite sensitivity and smoothness for rigorous analysis.