[Paper Review] Robustness Threats of Differential Privacy

Nurislam Tursynbek, Aleksandr Petiushko+1 more|arXiv (Cornell University)|Dec 14, 2020

Privacy-Preserving Technologies in Data57 references4 citations

TL;DR

This paper investigates the trade-off between differential privacy (DP) and model robustness in deep learning, demonstrating that DP-SGD training can reduce robustness to adversarial attacks and data corruptions despite enhancing privacy. Surprisingly, under certain conditions—particularly high gradient clipping bounds—DP can also improve robustness due to noise-induced optimization benefits.

ABSTRACT

Differential privacy (DP) is a gold-standard concept of measuring and guaranteeing privacy in data analysis. It is well-known that the cost of adding DP to deep learning model is its accuracy. However, it remains unclear how it affects robustness of the model. Standard neural networks are not robust to different input perturbations: either adversarial attacks or common corruptions. In this paper, we empirically observe an interesting trade-off between privacy and robustness of neural networks. We experimentally demonstrate that networks, trained with DP, in some settings might be even more vulnerable in comparison to non-private versions. To explore this, we extensively study different robustness measurements, including FGSM and PGD adversaries, distance to linear decision boundaries, curvature profile, and performance on a corrupted dataset. Finally, we study how the main ingredients of differentially private neural networks training, such as gradient clipping and noise addition, affect (decrease and increase) the robustness of the model.

Motivation & Objective

• To investigate whether training deep neural networks with differential privacy (DP-SGD) compromises their robustness against adversarial attacks and data corruptions.
• To analyze the impact of key DP-SGD components—gradient clipping and noise addition—on model robustness.
• To determine whether the privacy-robustness trade-off is fundamental or context-dependent, especially under varying hyperparameters.
• To explore conditions under which DP might simultaneously improve both privacy and robustness, challenging the assumption of inherent conflict.

Proposed method

• Empirically evaluate robustness using five metrics: FGSM and PGD attack success rates, number of iterations to adversarial misclassification, distance to decision boundaries in ℓ₂ and ℓ∞ norms, curvature profile, and performance on corrupted MNIST.
• Train convolutional neural networks on MNIST using DP-SGD with varying clipping bounds (C) and noise scales (σ), systematically measuring robustness and privacy trade-offs.
• Decompose the effects of gradient clipping and noise addition by analyzing their individual contributions to robustness changes.
• Use the DP-SGD algorithm with per-example gradient clipping and Gaussian noise injection to ensure (ε, δ)-differential privacy.
• Visualize decision boundaries via cross-sections to qualitatively compare geometric changes between standard SGD and DP-SGD models.
• Analyze the role of gradient noise (N(0, σ²C²I)) in optimization and its potential to improve generalization and robustness in deep networks.

Experimental results

Research questions

• RQ1 Does training with DP-SGD reduce the robustness of deep neural networks to adversarial examples and common data corruptions?
• RQ2 How do the core components of DP-SGD—gradient clipping and noise addition—affect model robustness?
• RQ3 Under what conditions, if any, can differential privacy improve model robustness rather than degrade it?
• RQ4 Is the privacy-robustness trade-off a fundamental limitation, or are there hyperparameter regimes where both privacy and robustness coexist?
• RQ5 What is the role of gradient noise magnitude (proportional to C²) in influencing robustness, and how does it interact with privacy budget?

Key findings

• DP-SGD training reduces robustness to FGSM and PGD attacks in most settings, with significant accuracy drops observed on adversarial examples.
• The distance to the nearest ℓ₂ and ℓ∞ decision boundary decreases under DP-SGD, indicating a more fragile decision boundary geometry.
• Curvature profiles of DP-SGD models show increased local curvature, suggesting a more sensitive and less stable decision surface.
• Performance on corrupted MNIST data degrades under DP-SGD, confirming reduced robustness to common corruptions.
• With sufficiently large clipping bounds (C = 3.0), a surprising 'plateau' in FGSM robustness appears, and further increasing C leads to higher robustness, indicating a non-monotonic trade-off.
• High gradient noise (from large C) can improve robustness, suggesting a beneficial optimization effect that may counteract privacy-induced fragility.