[论文解读] Quantifying and Mitigating Privacy Risks of Contrastive Learning

Xinlei He, Yang Zhang|arXiv (Cornell University)|Feb 8, 2021

Privacy-Preserving Technologies in Data参考文献 69被引用 6

一句话总结

本文首次对对比学习进行了隐私分析，揭示尽管由于过拟合程度降低，对比模型对成员推理攻击的抵抗力更强，但其表达性表征导致其对属性推理攻击的敏感度显著提高。为缓解此问题，作者提出了 Talos，一种基于对抗训练的隐私保护对比学习框架，通过抑制敏感属性泄露，有效降低属性推理风险，同时不损害模型效用或成员隐私。

ABSTRACT

Data is the key factor to drive the development of machine learning (ML) during the past decade. However, high-quality data, in particular labeled data, is often hard and expensive to collect. To leverage large-scale unlabeled data, self-supervised learning, represented by contrastive learning, is introduced. The objective of contrastive learning is to map different views derived from a training sample (e.g., through data augmentation) closer in their representation space, while different views derived from different samples more distant. In this way, a contrastive model learns to generate informative representations for data samples, which are then used to perform downstream ML tasks. Recent research has shown that machine learning models are vulnerable to various privacy attacks. However, most of the current efforts concentrate on models trained with supervised learning. Meanwhile, data samples' informative representations learned with contrastive learning may cause severe privacy risks as well. In this paper, we perform the first privacy analysis of contrastive learning through the lens of membership inference and attribute inference. Our experimental results show that contrastive models trained on image datasets are less vulnerable to membership inference attacks but more vulnerable to attribute inference attacks compared to supervised models. The former is due to the fact that contrastive models are less prone to overfitting, while the latter is caused by contrastive models' capability of representing data samples expressively. To remedy this situation, we propose the first privacy-preserving contrastive learning mechanism, Talos, relying on adversarial training. Empirical results show that Talos can successfully mitigate attribute inference risks for contrastive models while maintaining their membership privacy and model utility.

研究动机与目标

• 量化对比学习模型中的隐私风险，特别是针对成员推理攻击和属性推理攻击。
• 探究尽管对比模型相比监督模型过拟合程度更低，为何仍可能泄露敏感信息。
• 开发一种新型隐私保护机制，以减少属性推理风险，同时保持模型效用和成员隐私。
• 在标准图像数据集上，于现实攻击场景下评估所提方法的有效性。

提出的方法

• 适配现有的成员推理攻击和属性推理攻击方法——基于神经网络的、基于度量的以及仅标签的——以应用于对比模型。
• 在对比学习流程中引入对抗分类器，以在表征学习过程中抑制敏感属性的学习。
• 通过对抗训练联合训练对比模型和对抗分类器，其中分类器试图推断敏感属性，而编码器则被优化以最小化该能力。
• 使用数据增强生成对比学习的正样本和负样本视图，保持标准 SimCLR 框架，并采用 MoCo 风格队列以提高内存效率。
• 在预训练阶段应用隐私保护机制（Talos），确保表征不编码敏感属性。
• 通过攻击准确率、下游任务上的模型性能以及成员推理成功率，评估隐私与效用之间的权衡。

实验结果

研究问题

• RQ1 与监督模型相比，对比模型在成员推理攻击下的脆弱性如何？
• RQ2 尽管过拟合程度较低，为何对比模型对属性推理攻击更易受攻击？
• RQ3 对抗训练能否有效抑制对比表征中的敏感属性泄露？
• RQ4 所提出的隐私保护机制在多大程度上保持了模型效用和成员隐私？
• RQ5 该隐私缓解方法能否在不同图像数据集和模型架构上实现泛化？

主要发现

• 在 STL10 数据集上，使用 ResNet-50 的对比模型的成员推理攻击准确率为 0.620，显著低于对应监督模型的 0.810。
• 在 UTKFace 数据集上，使用 ResNet-18 的对比模型的属性推理攻击准确率达到 0.701，而监督模型仅为 0.422。
• 对属性推理攻击的脆弱性增加，归因于对比表征的表达性和信息丰富性，其保留了输入样本的详细信息。
• Talos 成功将属性推理攻击准确率降低至接近基线水平，同时在下游分类任务中保持高性能。
• Talos 有效保护了成员隐私，与标准对比学习相比，成员推理成功率无显著提升。
• 该方法在不同架构和数据集上表现出鲁棒性，证实其作为对比学习隐私保护方案的泛化能力。