[Paper Review] No Free Lunch in "Privacy for Free: How does Dataset Condensation Help Privacy"

Nicholas Carlini, Vitaly Feldman+1 more|arXiv (Cornell University)|Sep 29, 2022

Privacy-Preserving Technologies in Data4 citations

TL;DR

This paper challenges the claim in [DZL22] that dataset condensation (DC) significantly improves machine learning model privacy. It demonstrates that the empirical evaluation of the baseline is flawed—incorrectly measuring attack success on a biased subset—leading to a misreported 92.8% attack advantage instead of the correct 1.6%. The authors show that DC offers no statistically significant privacy improvement over random subsampling, and the theoretical analysis rests on unrealistic assumptions that trivially imply differential privacy, rendering the privacy claims invalid.

ABSTRACT

New methods designed to preserve data privacy require careful scrutiny. Failure to preserve privacy is hard to detect, and yet can lead to catastrophic results when a system implementing a ``privacy-preserving'' method is attacked. A recent work selected for an Outstanding Paper Award at ICML 2022 (Dong et al., 2022) claims that dataset condensation (DC) significantly improves data privacy when training machine learning models. This claim is supported by theoretical analysis of a specific dataset condensation technique and an empirical evaluation of resistance to some existing membership inference attacks. In this note we examine the claims in the work of Dong et al. (2022) and describe major flaws in the empirical evaluation of the method and its theoretical analysis. These flaws imply that their work does not provide statistically significant evidence that DC improves the privacy of training ML models over a naive baseline. Moreover, previously published results show that DP-SGD, the standard approach to privacy preserving ML, simultaneously gives better accuracy and achieves a (provably) lower membership attack success rate.

Motivation & Objective

• To critically evaluate the empirical and theoretical claims in [DZL22] that dataset condensation (DC) enhances model privacy.
• To expose methodological flaws in the baseline comparison used in [DZL22], particularly in how membership inference attack (MIA) success was measured.
• To demonstrate that the reported privacy improvement of DC over a naive baseline is not statistically significant due to incorrect evaluation setup.
• To challenge the theoretical privacy analysis in [DZL22], showing it relies on unrealistic assumptions that trivially imply differential privacy.
• To establish that DP-SGD remains superior in both accuracy and provable privacy, outperforming DC in privacy-utility trade-offs.

Proposed method

• Re-evaluates the membership inference attack (MIA) on the baseline scheme using the correct protocol: measuring attack success over the full dataset U, not a biased subset S ∪ S’.
• Corrects the baseline attack advantage from 92.8% to 1.6% by ensuring the base rate of membership is 1%, matching the actual training data fraction used.
• Uses a worst-case adversarial setting: trains models on two datasets differing by one example (D and D’), then tests if an adversary can distinguish them.
• Applies the distribution matching (DM) technique with random initialization and standard augmentations to assess detection capability of membership inference.
• Analyzes the theoretical privacy claims in [DZL22] by examining the assumptions, particularly Assumption 4.8, which models the output as an exponential mechanism.
• Shows that the theoretical analysis is uninformative because the transformation reduces to identity in high dimensions when initialized with a random subset or centered data.

Experimental results

Research questions

• RQ1 Does dataset condensation (DC) provide a statistically significant privacy advantage over a random subsampling baseline?
• RQ2 Is the empirical evaluation of the baseline in [DZL22] correctly configured to measure membership inference attack success?
• RQ3 Can a membership inference adversary distinguish between models trained on datasets differing by a single example when DC is applied?
• RQ4 Do the theoretical privacy guarantees in [DZL22] hold under realistic assumptions, particularly for the distribution matching (DM) method?
• RQ5 How does DC compare to established privacy-preserving methods like DP-SGD in terms of privacy and utility?

Key findings

• The baseline attack advantage reported in [DZL22] as 92.8% is incorrect; the correct value is 1.6%, due to an unfair evaluation setup using a subset S ∪ S’ with a 50% membership base rate.
• The actual attack advantage of DM (1.06 ± 1.20%) is not statistically distinguishable from the corrected baseline (1.6%), invalidating the claim that DC improves privacy.
• In a worst-case setting, an adversary can achieve 100% detection accuracy in distinguishing models trained on datasets differing by a single example, proving DM does not hide membership.
• The theoretical analysis in [DZL22] relies on Assumption 4.8, which models the model output as an exponential mechanism, trivially implying differential privacy and making the analysis uninformative.
• The transformation in the theoretical model reduces to identity in high dimensions when initialized with a random subset or centered data, meaning no meaningful generalization or privacy improvement occurs.
• DP-SGD achieves better accuracy and provably lower membership attack success rates than DC, making it a superior alternative for privacy-preserving machine learning.