[Paper Review] Systematic Evaluation of Privacy Risks of Machine Learning Models

Liwei Song, Prateek Mittal|arXiv (Cornell University)|Mar 24, 2020

Adversarial Robustness in Machine LearningComputer Science45 references84 citations

TL;DR

The paper critiques prior membership inference risk evaluations, introduces non-NN benchmark attacks, proposes a fine-grained privacy risk score, and shows defenses are less effective than claimed. It provides an evaluation protocol and public code.

ABSTRACT

Machine learning models are prone to memorizing sensitive data, making them vulnerable to membership inference attacks in which an adversary aims to guess if an input sample was used to train the model. In this paper, we show that prior work on membership inference attacks may severely underestimate the privacy risks by relying solely on training custom neural network classifiers to perform attacks and focusing only on the aggregate results over data samples, such as the attack accuracy. To overcome these limitations, we first propose to benchmark membership inference privacy risks by improving existing non-neural network based inference attacks and proposing a new inference attack method based on a modification of prediction entropy. We also propose benchmarks for defense mechanisms by accounting for adaptive adversaries with knowledge of the defense and also accounting for the trade-off between model accuracy and privacy risks. Using our benchmark attacks, we demonstrate that existing defense approaches are not as effective as previously reported. Next, we introduce a new approach for fine-grained privacy analysis by formulating and deriving a new metric called the privacy risk score. Our privacy risk score metric measures an individual sample's likelihood of being a training member, which allows an adversary to identify samples with high privacy risks and perform attacks with high confidence. We experimentally validate the effectiveness of the privacy risk score and demonstrate that the distribution of privacy risk score across individual samples is heterogeneous. Finally, we perform an in-depth investigation for understanding why certain samples have high privacy risks, including correlations with model sensitivity, generalization error, and feature embeddings. Our work emphasizes the importance of a systematic and rigorous evaluation of privacy risks of machine learning models.

Motivation & Objective

• Assess privacy risks from membership inference attacks beyond neural-network based attackers.
• Introduce non-NN benchmark attacks and a ground-truth-inspired entropy-based attack to measure privacy risks.
• Propose a fine-grained privacy risk score to evaluate per-sample risk.
• Evaluate existing defenses under adaptive/adversarial settings.
• Provide accessible benchmarks and code for reproducible privacy-risk assessment.

Proposed method

• Benchmark with non-NN based inference attacks including class-dependent thresholds and a modified prediction-entropy based attack.
• Introduce a new refined metric, modified prediction entropy (Mentr), to better capture ground-truth label information.
• Use shadow-training to set class-specific thresholds for metric-based attacks.
• Evaluate defenses under adaptive adversaries and compare against early stopping baselines.
• Propose and compute privacy risk scores for individual samples to reveal heterogeneity in risk.

Experimental results

Research questions

• RQ1 Do non-NN based attacks reveal higher membership inference risk than NN-based attacks on defended models?
• RQ2 How do class-specific thresholds and a modified entropy metric affect attack effectiveness?
• RQ3 Can per-sample privacy risk scores reveal heterogeneity in privacy risk across training samples?
• RQ4 Are existing defenses (e.g., adversarial regularization, MemGuard) robust under adaptive/adversarial evaluation?
• RQ5 How should privacy risk assessments be standardized to balance model accuracy and privacy?

Key findings

• Non-NN benchmark attacks significantly raise inferred privacy risk compared to prior NN-based evaluations (e.g., 58.6%–74.2% vs ~50%).
• Defense methods like adversarial regularization and MemGuard offer limited privacy protection under adaptive threats, not consistently outperforming early stopping.
• A modified prediction entropy (Mentr) attack outperforms standard entropy-based attacks.
• Privacy risk is heterogeneous across samples; the proposed privacy risk score can identify high-risk members for targeted membership inference.
• Per-sample risk analysis complements aggregate analyses to better understand privacy dynamics and guide defense evaluation.