[Paper Review] Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening

Jack Foster, Stefan Schoepf+1 more|arXiv (Cornell University)|Aug 15, 2023

Advanced Neural Network ApplicationsComputer Science3 citations

TL;DR

This paper proposes Selective Synaptic Dampening (SSD), a retraining-free, post-hoc machine unlearning method that selectively dampens model parameters disproportionately important to forgotten data using Fisher information matrix (FIM) analysis. SSD achieves unlearning performance competitive with retrain-based state-of-the-art methods while being orders of magnitude faster and requiring no long-term storage of training data.

ABSTRACT

Machine unlearning, the ability for a machine learning model to forget, is becoming increasingly important to comply with data privacy regulations, as well as to remove harmful, manipulated, or outdated information. The key challenge lies in forgetting specific information while protecting model performance on the remaining data. While current state-of-the-art methods perform well, they typically require some level of retraining over the retained data, in order to protect or restore model performance. This adds computational overhead and mandates that the training data remain available and accessible, which may not be feasible. In contrast, other methods employ a retrain-free paradigm, however, these approaches are prohibitively computationally expensive and do not perform on par with their retrain-based counterparts. We present Selective Synaptic Dampening (SSD), a novel two-step, post hoc, retrain-free approach to machine unlearning which is fast, performant, and does not require long-term storage of the training data. First, SSD uses the Fisher information matrix of the training and forgetting data to select parameters that are disproportionately important to the forget set. Second, SSD induces forgetting by dampening these parameters proportional to their relative importance to the forget set with respect to the wider training data. We evaluate our method against several existing unlearning methods in a range of experiments using ResNet18 and Vision Transformer. Results show that the performance of SSD is competitive with retrain-based post hoc methods, demonstrating the viability of retrain-free post hoc unlearning approaches.

Motivation & Objective

• To address the growing need for machine unlearning in compliance with data privacy regulations like GDPR.
• To develop a retraining-free unlearning method that preserves model performance on retained data without requiring access to original training data after initial processing.
• To identify and selectively modify only the parameters most specialized to forgotten samples, minimizing impact on generalization.
• To achieve unlearning efficiency (speed) and performance parity with retraining-based methods while avoiding the storage and computational overhead of full retraining.

Proposed method

• SSD computes the diagonal of the Fisher Information Matrix (FIM) for both the forgotten and retained data to quantify parameter importance.
• It identifies parameters that are significantly more important to the forgotten data than to the retained data, indicating specialization.
• SSD dampens these specialized parameters proportionally to their relative importance to the forgotten set, using a damping factor controlled by hyperparameters α and λ.
• The method operates post-hoc, requiring only a single access to the training data to compute FIM, after which the data can be discarded.
• The approach leverages the principle that overparameterized models memorize training data, so targeting memorized parameters enables selective forgetting.
• The damping process is applied only to a small subset of parameters, preserving model consistency and generalization on retained data.

Experimental results

Research questions

• RQ1 Can a retraining-free unlearning method achieve performance comparable to retraining-based state-of-the-art methods?
• RQ2 Can selective parameter dampening based on Fisher information effectively induce forgetting without degrading model performance on retained data?
• RQ3 Is it possible to achieve fast unlearning without storing the original training data after FIM computation?
• RQ4 How does the performance of SSD compare to existing retrain-free and retraining-based unlearning methods across different forgetting scenarios?
• RQ5 What is the impact of hyperparameter selection (α and λ) on unlearning efficacy and model degradation over repeated forgetting requests?

Key findings

• SSD reduced the membership inference attack (MIA) accuracy from 93% (baseline ResNet) and 94% (ViT) to approximately 2% on CIFAR100, matching the performance of full retraining.
• On ResNet, the accuracy on the retained set (Dr) dropped only 2% after unlearning, while ViT showed a negligible improvement, indicating strong performance preservation.
• SSD outperformed the prior retrain-free method Fisher Forgetting by orders of magnitude in speed while achieving superior unlearning performance.
• In subclass unlearning on CIFAR20, SSD achieved MIA and Dr performance comparable to retrained models, unlike methods like Amnesiac and Bad Teacher that showed artificially low MIA scores.
• For random sample unlearning on CIFAR10, SSD matched retraining performance in both speed and effectiveness for both ResNet and ViT.
• SSD was the strongest-performing method on average in terms of similarity to the fully retrained model across all benchmarks.