[论文解读] Differential Privacy Has Disparate Impact on Model Accuracy

Eugene Bagdasaryan, Vitaly Shmatikov|arXiv (Cornell University)|May 28, 2019

Privacy-Preserving Technologies in Data参考文献 38被引用 77

一句话总结

论文表明 DP-SGD 在准确性上的退化不均衡，较多损害被代表性不足的子群体和复杂数据，而对代表性较强的群体影响较小，并且 DP 可能放大现有的不公平性。

ABSTRACT

Differential privacy (DP) is a popular mechanism for training machine learning models with bounded leakage about the presence of specific points in the training data. The cost of differential privacy is a reduction in the model's accuracy. We demonstrate that in the neural networks trained using differentially private stochastic gradient descent (DP-SGD), this cost is not borne equally: accuracy of DP models drops much more for the underrepresented classes and subgroups. For example, a gender classification model trained using DP-SGD exhibits much lower accuracy for black faces than for white faces. Critically, this gap is bigger in the DP model than in the non-DP model, i.e., if the original model is unfair, the unfairness becomes worse once DP is applied. We demonstrate this effect for a variety of tasks and models, including sentiment analysis of text and image classification. We then explain why DP training mechanisms such as gradient clipping and noise addition have disproportionate effect on the underrepresented and more complex subgroups, resulting in a disparate reduction of model accuracy.

研究动机与目标

• Motivate investigation of how differential privacy via DP-SGD affects accuracy across subgroups and data complexity.
• Demonstrate empirically that DP training disproportionately harms underrepresented classes and subgroups across multiple tasks.
• Explain mechanisms (gradient clipping and noise) that amplify bias against underrepresented data.
• Highlight the interaction between pre-existing model unfairness and privacy-induced accuracy degradation.

提出的方法

• Implement DP-SGD with gradient clipping and Gaussian noise using the moments accountant for privacy tracking.
• Conduct extensive experiments across tasks: gender/age classification on faces, sentiment analysis of tweets, species classification, and federated language modeling.
• Vary hyperparameters (clipping bound S, noise scale, batch size, epochs) to study their impact on subgroup accuracy.
• Analyze gradient norms per class and the effect of clipping/noise on underrepresented groups.
• Use MNIST-based experiments to illustrate how hyperparameters influence disparate impact.
• Measure privacy loss using Rényi DP to keep epsilon under targeted levels.

实验结果

研究问题

• RQ1 Does DP-SGD cause larger accuracy degradation for underrepresented subgroups compared to well-represented ones?
• RQ2 Do gradient clipping and noise in DP-SGD disproportionately affect underrepresented data due to gradient size disparities?
• RQ3 How do DP settings (S, noise, batch size, epochs) influence the fairness of accuracy across subgroups?
• RQ4 Is existing unfairness in a non-DP model amplified by applying differential privacy?
• RQ5 Do effects of DP on accuracy hold across diverse tasks (vision, NLP, federated learning)?

主要发现

• DP 模型在 DP-SGD 下对较深色皮肤的人脸的准确性下降比对较浅色人脸的更明显。
• DP 训练在被代表性不足的子群体和数据较复杂的子群体上的准确性下降，优于非 DP 模型。
• 在 DP 下，被代表性不足组与代表性较强组之间的差距常常扩大，体现为“poor get poorer”现象。
• 梯度裁剪和引入的噪声会不成比例地减少来自被代表性不足数据的学习更新，从而在多项任务（人脸属性、情感、物种、联邦语言学习）中放大偏差。
• 在类似 MNIST 的实验中，低频类别、样本数量较少的类别在 DP 下的准确性退化要大得多，即使总体 epsilon 保持在 10 以下。