[论文解读] Efficient Attribute Unlearning: Towards Selective Removal of Input Attributes from Feature Representations

Tao Guo, Song Guo+3 more|arXiv (Cornell University)|Feb 27, 2022

Face recognition and analysis被引用 7

一句话总结

本文提出了一种名为'属性遗忘'的新机器遗忘范式，该范式在训练过程中通过互信息引导，选择性地从学习到的特征表示中移除特定输入属性（如眼镜或年龄）。该方法在实现高效率遗忘的同时，性能下降极小，从而在保留任务相关特征的前提下，实现了隐私保护的模型训练。

ABSTRACT

Recently, the enactment of privacy regulations has promoted the rise of the machine unlearning paradigm. Existing studies of machine unlearning mainly focus on sample-wise unlearning, such that a learnt model will not expose user's privacy at the sample level. Yet we argue that such ability of selective removal should also be presented at the attribute level, especially for the attributes irrelevant to the main task, e.g., whether a person recognized in a face recognition system wears glasses or the age range of that person. Through a comprehensive literature review, it is found that existing studies on attribute-related problems like fairness and de-biasing learning cannot address the above concerns properly. To bridge this gap, we propose a paradigm of selectively removing input attributes from feature representations which we name `attribute unlearning'. In this paradigm, certain attributes will be accurately captured and detached from the learned feature representations at the stage of training, according to their mutual information. The particular attributes will be progressively eliminated along with the training procedure towards convergence, while the rest of attributes related to the main task are preserved for achieving competitive model performance. Considering the computational complexity during the training process, we not only give a theoretically approximate training method, but also propose an acceleration scheme to speed up the training process. We validate our method by spanning several datasets and models and demonstrate that our design can preserve model fidelity and reach prevailing unlearning efficacy with high efficiency. The proposed unlearning paradigm builds a foundation for future machine unlearning system and will become an essential component of the latest privacy-related legislation.

研究动机与目标

• 为解决现有机器遗忘方法仅关注样本级遗忘，而未涉及属性级选择性移除的问题。
• 实现在不损害主要任务性能的前提下，精确分离并移除学习特征表示中无关或敏感的属性（如面部眼镜、年龄）。
• 开发一种理论基础扎实、计算高效的训练方法，支持在模型训练过程中逐步遗忘特定属性。
• 确保属性遗忘在可扩展性方面与现实世界隐私法规（如GDPR和CCPA）兼容。

提出的方法

• 该方法利用输入属性与特征表示之间的互信息，识别并隔离需遗忘的敏感属性。
• 引入表示解耦模块，在训练过程中逐步将目标属性从主特征空间中分离。
• 提出一种近似技术，通过高效估算互信息损失来降低计算复杂度，从而实现可扩展的训练。
• 该方法可无缝集成到标准深度学习流水线中，仅需极少的架构修改，支持端到端训练。
• 通过专用损失函数优化训练过程，以促进属性解耦，同时保留与任务相关的特征。
• 引入加速方案以加快收敛速度，尤其在大规模数据集上具有显著优势。

实验结果

研究问题

• RQ1 我们能否在不降低主任务模型性能的前提下，从学习到的特征表示中选择性地移除特定输入属性？
• RQ2 如何利用输入属性与特征之间的互信息来引导遗忘过程？
• RQ3 一种高效且可扩展的训练方法应如何设计，以实现低计算开销的渐进式属性遗忘？
• RQ4 属性遗忘在消除敏感属性的同时，能在多大程度上保持模型保真度？

主要发现

• 所提出的属性遗忘方法在训练过程中能以高精度成功从特征表示中解耦目标属性（如眼镜、年龄）。
• 该方法在主分类任务上保持了具有竞争力的模型性能，遗忘后准确率下降极小。
• 互信息损失计算的近似方法显著减少了训练时间，同时保持了遗忘的有效性。
• 在多个数据集和模型上的大量实验验证了该方法在真实场景中的鲁棒性与可扩展性。
• 与现有公平性与去偏方法相比，该方法在显式移除特定属性方面表现更优，而后者无法保证此类效果。
• 该框架为满足隐私法规中关于从模型表示中选择性移除个人属性的要求提供了基础支持。