[论文解读] The Unintended Consequences of Overfitting: Training Data Inference Attacks.

Samuel Yeom, Matt Fredrikson+1 more|arXiv (Cornell University)|Sep 5, 2017

Adversarial Robustness in Machine Learning被引用 20

一句话总结

本文研究了机器学习模型中的过拟合与特征影响如何使成员推理攻击和模型反演攻击成为可能。研究证明，过拟合足以实现成员推理攻击，并且在特定影响条件下可实现模型反演攻击，揭示了攻击类型之间的深层联系，并识别出除过拟合以外的其他导致隐私泄露的因素。

ABSTRACT

Machine learning algorithms that are applied to sensitive data pose a distinct threat to privacy. A growing body of prior work demonstrates that models produced by these algorithms may leak specific private information in the training data to an attacker, either through their structure or their observable behavior. However, the underlying cause of this privacy risk is not well understood beyond a handful of anecdotal accounts that suggest overfitting and influence might play a role. This paper examines the effect that overfitting and influence have on the ability of an attacker to learn information about training data from machine learning models, either through training set membership inference or model inversion attacks. Using both formal and empirical analyses, we illustrate a clear relationship between these factors and the privacy risk that arises in several popular machine learning algorithms. We find that overfitting is sufficient to allow an attacker to perform membership inference, and when certain conditions on the influence of certain features are present, model inversion attacks. Interestingly, our formal analysis also shows that overfitting is not necessary for these attacks, and begins to shed light on what other factors may be in play. Finally, we explore the connection between two types of attack, membership inference and model inversion, and show that there are deep connections between the two that lead to effective new attacks.

研究动机与目标

• 理解在敏感数据上训练的机器学习模型中隐私泄露的根本原因。
• 研究过拟合与特征影响在促成成员推理和模型反演攻击中的作用。
• 确定过拟合是否是此类攻击的必要条件，或是否存在其他促成隐私风险的因素。
• 探索成员推理与模型反演攻击之间的联系，并识别协同攻击策略。

提出的方法

• 对机器学习模型中模型过拟合、特征影响与隐私泄露之间关系的正式分析。
• 在多种主流机器学习算法上对成员推理和模型反演攻击进行实证评估。
• 识别出能够实现成功模型反演攻击的特定特征影响条件。
• 在不同过拟合程度和影响水平下比较攻击效果，以隔离因果因素。
• 开发一个统一框架，用于分析并连接成员推理与模型反演攻击。

实验结果

研究问题

• RQ1 过拟合在多大程度上能够使成员推理攻击在机器学习模型上成为可能？
• RQ2 在何种特征影响条件下，模型反演攻击变得可行？
• RQ3 过拟合是否是通过成员推理或模型反演导致隐私泄露的必要条件？
• RQ4 哪些结构或行为特性将成员推理与模型反演攻击联系在一起？
• RQ5 能否利用这两种攻击类型之间的关联来设计更有效的隐私攻击？

主要发现

• 即使缺乏其他可识别因素，过拟合也足以实现成员推理攻击。
• 当满足特定特征影响条件时，模型反演攻击是可行的，表明影响动态在其中起着关键作用。
• 过拟合并非隐私攻击的必要条件，表明其他模型特性也会导致隐私泄露。
• 成员推理与模型反演攻击之间存在深层次的结构与行为关联，可支持混合攻击策略。
• 正式分析表明，隐私风险不仅与记忆化有关，还与特征如何塑造模型行为和输出密切相关。