[Paper Review] Does Label Differential Privacy Prevent Label Inference Attacks?

Ruihan Wu, Jin Zhou+2 more|arXiv (Cornell University)|Feb 25, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper resolves the paradox that label-differentially private (label-DP) models can still be vulnerable to label inference attacks (LIAs) by redefining success as advantage over the Bayes classifier, not absolute accuracy. It proves that label-DP with parameter ε limits this advantage to $1 - \frac{2}{1 + e^{\varepsilon}}$, showing that even high training accuracy does not imply private label leakage when ε is small, and validates this bound empirically on synthetic and real-world datasets including Criteo 1TB Click Logs.

ABSTRACT

Label differential privacy (label-DP) is a popular framework for training private ML models on datasets with public features and sensitive private labels. Despite its rigorous privacy guarantee, it has been observed that in practice label-DP does not preclude label inference attacks (LIAs): Models trained with label-DP can be evaluated on the public training features to recover, with high accuracy, the very private labels that it was designed to protect. In this work, we argue that this phenomenon is not paradoxical and that label-DP is designed to limit the advantage of an LIA adversary compared to predicting training labels using the Bayes classifier. At label-DP $ε=0$ this advantage is zero, hence the optimal attack is to predict according to the Bayes classifier and is independent of the training labels. Our bound shows the semantic protection conferred by label-DP and gives guidelines on how to choose $\varepsilon$ to limit the threat of LIAs below a certain level. Finally, we empirically demonstrate that our result closely captures the behavior of simulated attacks on both synthetic and real world datasets.

Motivation & Objective

• To resolve the apparent paradox that label-DP models can still be attacked via label inference despite strong privacy guarantees.
• To reframe the evaluation of label inference attacks (LIAs) by measuring adversary advantage over the Bayes classifier, rather than absolute prediction accuracy.
• To derive a tight upper bound on the advantage of any LIA adversary under label-DP, linking it directly to the privacy parameter ε.
• To empirically validate the theoretical bound on both synthetic and real-world datasets, including the highly skewed Criteo 1TB Click Logs dataset.
• To provide practical guidelines for selecting ε to limit LIA risk below a desired threshold in real-world applications.

Proposed method

• Define the success of a label inference attack (LIA) not by absolute accuracy, but by its advantage over the Bayes classifier, which is independent of training labels.
• Formally prove that an ε-label-DP learner limits the LIA adversary’s advantage to $1 - \frac{2}{1 + e^{\varepsilon}}$, which decreases as ε decreases.
• Use a weighted attack utility function to account for class imbalance and label value differences, ensuring fairness in evaluation.
• Estimate the expected attack utility (EAU) and lower-bound expected attack utility (L-EAU) using model training accuracy and test performance on real datasets.
• Validate the theoretical bound on both synthetic data (where the Bayes classifier is known) and real-world data (Criteo 1TB Click Logs) with varying ε values.
• Apply label-DP accounting and evaluate the gap between theoretical upper bound and empirical advantage to assess tightness and potential for improved attacks.

Experimental results

Research questions

• RQ1 Why do label inference attacks (LIAs) achieve high accuracy on label-DP models despite strong privacy guarantees?
• RQ2 Is label-DP truly ineffective against LIAs, or is the evaluation metric misleading?
• RQ3 Can the advantage of an LIA adversary over the Bayes classifier be bounded under label-DP, and if so, how does it relate to ε?
• RQ4 Does high training accuracy in label-DP models necessarily imply private label leakage?
• RQ5 How tight is the theoretical upper bound on LIA advantage, and how does it perform on real-world, noisy, and imbalanced datasets?

Key findings

• The theoretical upper bound on LIA advantage under ε-label-DP is $1 - \frac{2}{1 + e^{\varepsilon}}$, which decreases with smaller ε and ensures semantic privacy.
• On the synthetic dataset, the theoretical upper bound tightly dominates the empirical advantage of the simple prediction attack (SPA) across both small and large ε values.
• On the Criteo 1TB Click Logs dataset, the SPA attack achieves ~97% label inference accuracy but has negative advantage for ε ≤ 2, indicating it performs worse than the Bayes classifier.
• Despite high training accuracy, label-DP with small ε does not reveal private labels beyond what is implied by the Bayes classifier, resolving the apparent paradox.
• The empirical gap between the theoretical upper bound and observed advantage suggests room for tighter label-DP accounting or more effective LIAs.
• The study shows that label-DP is not compromised by LIAs when evaluated via the correct metric—adversary advantage over the Bayes classifier—providing a principled privacy-utility calibration framework.