[论文解读] Better accuracy with quantified privacy: representations learned via reconstructive adversarial network

Sicong Liu, Anshumali Shrivastava+2 more|arXiv (Cornell University)|Jan 25, 2019

Privacy-Preserving Technologies in Data参考文献 24被引用 13

一句话总结

该论文提出RAN，一种新颖的深度学习框架，通过对抗性训练联合优化表示学习中的隐私与效用。通过三部分架构——编码器、分类器和解码器——联合最小化分类误差（效用）和重构误差（隐私），RAN 学习到的特征在下游任务中具有高度准确性，同时对原始数据重构具有强抵抗能力，在实现比基线模型更高准确率的同时，通过对抗性正则化隐式提升隐私保护。

ABSTRACT

The remarkable success of machine learning, especially deep learning, has produced a variety of cloud-based services for mobile users. Such services require an end user to send data to the service provider, which presents a serious challenge to end-user privacy. To address this concern, prior works either add noise to the data or send features extracted from the raw data. They struggle to balance between the utility and privacy because added noise reduces utility and raw data can be reconstructed from extracted features. This work represents a methodical departure from prior works: we balance between a measure of privacy and another of utility by leveraging adversarial learning to find a sweeter tradeoff. We design an encoder that optimizes against the reconstruction error (a measure of privacy), adversarially by a Decoder, and the inference accuracy (a measure of utility) by a Classifier. The result is RAN, a novel deep model with a new training algorithm that automatically extracts features for classification that are both private and useful. It turns out that adversarially forcing the extracted features to only conveys the intended information required by classification leads to an implicit regularization leading to better classification accuracy than the original model which completely ignores privacy. Thus, we achieve better privacy with better utility, a surprising possibility in machine learning! We conducted extensive experiments on five popular datasets over four training schemes, and demonstrate the superiority of RAN compared with existing alternatives.

研究动机与目标

• 解决云基机器学习服务中数据隐私与模型效用之间的矛盾。
• 开发一种通过重构误差量化隐私，同时最大化分类准确率的方法。
• 设计一个端到端深度模型，通过对抗性训练同时优化隐私与效用。
• 证明对抗性正则化可隐式去除冗余信息，从而提升泛化能力与准确率。

提出的方法

• RAN 采用三部分架构：编码器用于特征提取，分类器用于效用（判别式）学习，解码器用于隐私（重构）学习。
• 编码器通过对抗性方式训练以对抗解码器，从而最大化重构误差，提升隐私保护。
• 分类器在编码特征上训练以最小化分类误差，确保模型效用。
• 提出一种新颖的三阶段训练算法，交替进行判别式学习、生成式学习（解码器）以及所有组件的对抗性微调。
• 模型使用反向传播与双梯度：一个来自分类器（用于效用），一个来自解码器（用于隐私），引导特征空间的优化。
• 隐私通过从编码特征重构原始数据的误差来量化，误差越高表示隐私保护越好。

实验结果

研究问题

• RQ1 对抗性训练能否同时提升表示学习中的模型效用与隐私？
• RQ2 强制模型抵抗原始数据重构是否能带来更好的泛化能力与更高的分类准确率？
• RQ3 单一端到端深度模型能否在隐私-效用权衡上优于现有分别应用噪声或特征提取的方法？
• RQ4 对抗性隐私约束带来的隐式正则化如何影响特征质量与模型泛化能力？

主要发现

• RAN 在分类准确率上优于忽略隐私的原始模型，表明隐私约束可提升泛化能力。
• 对抗性训练过程隐式去除了特征中的冗余信息，从而提升模型鲁棒性与性能。
• RAN 在隐私与效用方面均优于现有方法（如噪声特征提取与差分隐私），且分类准确率高于噪声基线模型。
• 在五个基准数据集（MNIST、CIFAR-10、ImageNet、Ubisound、HAR）上的大量实验验证了RAN在隐私与效用平衡方面的优越性。
• 重构误差度量能有效量化隐私，对抗性训练针对该误差可生成难以逆向工程的特征。