[Paper Review] Evaluating Differentially Private Machine Learning in Practice

Bargav Jayaraman, David Evans|arXiv (Cornell University)|Feb 24, 2019

Privacy-Preserving Technologies in Data80 references85 citations

TL;DR

The paper empirically evaluates differential privacy variants (AC, zCDP, RDP) in gradient-perturbed ML, measuring actual privacy leakage via membership inference and attribute inference attacks across logistic regression and neural networks.

ABSTRACT

Differential privacy is a strong notion for privacy that can be used to prove formal guarantees, in terms of a privacy budget, $ε$, about how much information is leaked by a mechanism. However, implementations of privacy-preserving machine learning often select large values of $ε$ in order to get acceptable utility of the model, with little understanding of the impact of such choices on meaningful privacy. Moreover, in scenarios where iterative learning procedures are used, differential privacy variants that offer tighter analyses are used which appear to reduce the needed privacy budget but present poorly understood trade-offs between privacy and utility. In this paper, we quantify the impact of these choices on privacy in experiments with logistic regression and neural network models. Our main finding is that there is a huge gap between the upper bounds on privacy loss that can be guaranteed, even with advanced mechanisms, and the effective privacy loss that can be measured using current inference attacks. Current mechanisms for differentially private machine learning rarely offer acceptable utility-privacy trade-offs with guarantees for complex learning tasks: settings that provide limited accuracy loss provide meaningless privacy guarantees, and settings that provide strong privacy guarantees result in useless models. Code for the experiments can be found here: https://github.com/bargavj/EvaluatingDPML

Motivation & Objective

• Assess how different epsilon settings affect privacy-utility trade-offs in DP ML.
• Compare how advanced composition, zCDP, and RDP influence cumulative privacy loss in practice.
• Quantify actual privacy leakage via membership inference and attribute inference attacks.
• Evaluate DP ML implementations on logistic regression and neural networks to identify practical risks.

Proposed method

• Review differential privacy definitions and variants (AC, zCDP, RDP) and their theoretical composition properties.
• Focus on gradient perturbation as the DP mechanism for ML, with gradient clipping to bound sensitivity.
• Use moments accountant / equivalent analyses to bound cumulative privacy loss under different DP notions.
• Empirically evaluate privacy leakage using membership inference and attribute inference attacks on DL and LR models.
• Apply DP mechanisms to ERM-based learning tasks and non-convex deep learning settings to study utility impacts.

Experimental results

Research questions

• RQ1 How do different epsilon values and DP variants (AC, zCDP, RDP) affect the privacy-utility trade-off in DP ML?
• RQ2 What is the practical gap between formal DP guarantees and actual privacy leakage observed through attacks?
• RQ3 How do DP variants perform in terms of leakage when applied to logistic regression and neural networks?
• RQ4 Do tighter composition analyses translate to meaningful privacy protection in real-world ML tasks?

Key findings

• There is a huge gulf between the upper bounds guaranteed by DP variants and what can be observed by known attacks.
• For acceptable utility levels, the formal guarantees are essentially meaningless regardless of the DP variant used.
• Observed leakage from membership and attribute inference attacks remains relatively low even under leakage-prone settings.
• Sensitivity to epsilon varies across DP variants, with trade-offs between utility and leakage that are task- and model-dependent.
• Gradient perturbation with clipping is a practical approach for DP in deep learning, but composition increases the privacy budget over many iterations.