[Paper Review] Auditing Differentially Private Machine Learning: How Private is Private SGD?

Matthew Jagielski, Jonathan Ullman+1 more|arXiv (Cornell University)|Jun 13, 2020

Privacy-Preserving Technologies in Data17 references73 citations

TL;DR

The paper empirically audits the privacy of DP-SGD using novel data poisoning attacks, deriving lower bounds on the privacy parameter ε that are significantly tighter than prior practical methods and near theoretical worst-case bounds.

ABSTRACT

We investigate whether Differentially Private SGD offers better privacy in practice than what is guaranteed by its state-of-the-art analysis. We do so via novel data poisoning attacks, which we show correspond to realistic privacy attacks. While previous work (Ma et al., arXiv 2019) proposed this connection between differential privacy and data poisoning as a defense against data poisoning, our use as a tool for understanding the privacy of a specific mechanism is new. More generally, our work takes a quantitative, empirical approach to understanding the privacy afforded by specific implementations of differentially private algorithms that we believe has the potential to complement and influence analytical work on differential privacy.

Motivation & Objective

• Motivation: bridge the gap between worst-case DP guarantees and practical privacy in DP-SGD.
• Goal: develop data-poisoning based audits to obtain lower bounds on ε for DP-SGD.
• Aim: assess how private DP-SGD is in practice and explore the usefulness of empirical auditing in differential privacy.
• Scope: evaluate on TensorFlow Privacy implementations with standard datasets and hyperparameters.

Proposed method

• Define poisoning-based audits that compare distributions of A(D0) vs A(D1) to bound ε via ε_LB = ln(p0/p1)/k.
• Develop Clopper–Pearson based confidence bounds to provide probabilistic guarantees on ε_LB.
• Design novel clipping-aware poisoning attacks (ClipBKD) that minimize gradient variance in the direction of least variance to survive gradient clipping.
• Compare attacks against DP-SGD with standard membership inference attacks across multiple datasets.
• Use two datasets (FMNIST, CIFAR10, Purchase-100) and two model types (logistic regression and 2-layer neural networks) under DP-SGD.
• Instantiate the approach on the TensorFlow Privacy DP-SGD implementation and report empirical ε_LB values.

Experimental results

Research questions

• RQ1 How close are practical ε values achieved by DP-SGD to its theoretical worst-case bounds?
• RQ2 Can data-poisoning based audits provide tighter lower bounds on ε than existing privacy attacks such as membership inference?
• RQ3 Does clipping in DP-SGD enable more robust poisoning attacks, and how can attacks exploit the direction of least variance?
• RQ4 How do dataset, model type, and DP-SGD hyperparameters affect the measured ε_LB in practice?
• RQ5 What insights can empirical auditing provide to complement theoretical DP analyses?

Key findings

• ClipBKD and clipping-aware poisoning attacks significantly outperform membership inference attacks in producing ε_LB across datasets and models.
• ClipBKD yields ε_LB values that are 2.5x–1500x higher than MI, and can approach the theoretical ε_th upper bound (e.g., ε_th = 4 yields ε_LB = 0.46 on Purchase-100 with 2-layer nets).
• Standard backdoor attacks often fail to produce positive ε_LB on CIFAR10 due to dataset/model factors, whereas ClipBKD consistently improves over standard backdoors.
• Sensitivity analyses show ε_LB increases with higher ε_th (lower noise), and is highly affected by initialization randomness and clipping norm; fixed initialization often yields higher ε_LB.
• Across FMNIST, CIFAR10, and P100, DP-SGD privacy measured by ClipBKD is substantially below the worst-case theoretical bounds, indicating real-world privacy can be weaker than formal guarantees suggest but still far from trivial.
• Training accuracy remains high (96–98%) across experiments, indicating privacy leaks can be quantified without sacrificing utility.