[论文解读] Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning

Milad Nasr, Shuang Song+3 more|arXiv (Cornell University)|Jan 11, 2021

Privacy-Preserving Technologies in Data被引用 7

一句话总结

本文提出一种在差分隐私（DP）机器学习中实例化假设敌手的方法，以建立隐私泄露的紧致下界。通过模拟从仅黑盒模型查询到完整梯度污染的各类敌手能力，表明当敌手具有最大访问权限时，DP-SGD的理论上限是紧致的，但在施加实际约束时，实际隐私保护显著更强。

ABSTRACT

Differentially private (DP) machine learning allows us to train models on private data while limiting data leakage. DP formalizes this data leakage through a cryptographic game, where an adversary must predict if a model was trained on a dataset D, or a dataset D' that differs in just one example.If observing the training algorithm does not meaningfully increase the adversary's odds of successfully guessing which dataset the model was trained on, then the algorithm is said to be differentially private. Hence, the purpose of privacy analysis is to upper bound the probability that any adversary could successfully guess which dataset the model was trained on.In our paper, we instantiate this hypothetical adversary in order to establish lower bounds on the probability that this distinguishing game can be won. We use this adversary to evaluate the importance of the adversary capabilities allowed in the privacy analysis of DP training algorithms.For DP-SGD, the most common method for training neural networks with differential privacy, our lower bounds are tight and match the theoretical upper bound. This implies that in order to prove better upper bounds, it will be necessary to make use of additional assumptions. Fortunately, we find that our attacks are significantly weaker when additional (realistic)restrictions are put in place on the adversary's capabilities.Thus, in the practical setting common to many real-world deployments, there is a gap between our lower bounds and the upper bounds provided by the analysis: differential privacy is conservative and adversaries may not be able to leak as much information as suggested by the theoretical bound.

研究动机与目标

• 探究理论差分隐私（DP）上界与实际中隐私泄露之间的差距。
• 评估敌手能力对DP机器学习中隐私攻击有效性的影响。
• 确定当前DP-SGD分析是否紧致，或是否可通过附加假设得到改进。
• 评估对敌手施加实际限制后，理论上的上界与经验下界之间是否存在显著差距。
• 通过识别哪些假设能带来最有益的隐私保障，为未来理论与应用研究提供指导。

提出的方法

• 从仅观察模型预测的黑盒敌手到可访问中间梯度和训练更新的白盒敌手，实例化一系列敌手。
• 构建两个数据集 D 和 D′，二者仅在单个样本上不同，以模拟DP区分游戏。
• 设计两种攻击算法：一种用于生成敌手数据集，另一种用于预测已训练模型是基于哪个数据集训练的。
• 在各种威胁模型下测量经验隐私损失（ε），并与DP分析得出的理论上限进行比较。
• 以DP-SGD为主要案例研究，应用矩会计（Moments Accountant）进行严格的隐私会计。
• 在MNIST和CIFAR-10等标准数据集上评估攻击，以评估威胁模型的泛化性和现实性。

Figure 2 : Our attack process. Our first algorithm, the crafter constructs two datasets $D$ and $D^{\prime}$ differing in one example. The model trainer then (independently of the adversary) trains a model on one of these two datasets. Our second algorithm, the distinguisher then guesses which datas

实验结果

研究问题

• RQ1 当敌手可完全访问训练中间结果时，DP-SGD的理论隐私泄露上界有多紧？
• RQ2 当敌手被限制为仅观察最终模型输出（即黑盒访问）时，实际隐私泄露水平如何？
• RQ3 通过限制敌手能力，能否显著缩小理论上的上界与经验下界之间的差距？
• RQ4 关于敌手访问权限的哪些假设（例如梯度污染与模型查询）对隐私泄露影响最大？
• RQ5 是否存在实际威胁模型，使得实际提供的隐私超过理论上的上界，表明当前DP分析存在过度保守？

主要发现

• 当敌手可完全访问训练梯度和中间模型更新时，DP-SGD的隐私损失（ε）理论上限是紧致的。
• 当敌手仅能观察最终模型预测（即黑盒访问）时，实际隐私泄露远低于理论上的上限。
• 在实际设置中，理论上的上界与经验下界之间存在显著差距，表明DP-SGD在实践中具有较强的保守性。
• 修改训练梯度的攻击仅比在真实数据集（如CIFAR-10）上操作的攻击略为有效，表明病态数据构造不会带来不成比例的优势。
• 将敌手限制为仅能修改训练样本（而非梯度）是改进隐私保障的有前途方向，且无需改变算法。
• 当前DP-SGD的矩会计分析是紧致的，意味着进一步改进理论边界需要算法变更或对敌手能力施加更强假设。