[Paper Review] Exploiting Machine Unlearning for Backdoor Attacks in Deep Learning System

Peixin Zhang, Jun Sun+2 more|arXiv (Cornell University)|Sep 12, 2023

Adversarial Robustness in Machine LearningComputer Science3 citations

TL;DR

This paper proposes BAU, a novel black-box backdoor attack that exploits machine unlearning to covertly implant a backdoor into a deep learning model. By injecting poison and mitigation data into the training set and then submitting unlearning requests for the mitigation samples, the attack gradually activates the backdoor, evading detection while significantly increasing the success rate of malicious predictions after unlearning.

ABSTRACT

In recent years, the security issues of artificial intelligence have become increasingly prominent due to the rapid development of deep learning research and applications. Backdoor attack is an attack targeting the vulnerability of deep learning models, where hidden backdoors are activated by triggers embedded by the attacker, thereby outputting malicious predictions that may not align with the intended output for a given input. In this work, we propose a novel black-box backdoor attack based on machine unlearning. The attacker first augments the training set with carefully designed samples, including poison and mitigation data, to train a `benign' model. Then, the attacker posts unlearning requests for the mitigation samples to remove the impact of relevant data on the model, gradually activating the hidden backdoor. Since backdoors are implanted during the iterative unlearning process, it significantly increases the computational overhead of existing defense methods for backdoor detection or mitigation. To address this new security threat, we proposes two methods for detecting or mitigating such malicious unlearning requests. We conduct the experiment in both exact unlearning and approximate unlearning (i.e., SISA) settings. Experimental results indicate that: 1) our attack approach can successfully implant backdoor into the model, and sharding increases the difficult of attack; 2) our detection algorithms are effective in identifying the mitigation samples, while sharding reduces the effectiveness of our detection algorithms.

Motivation & Objective

• To investigate the security risks introduced by machine unlearning mechanisms in deep learning systems.
• To demonstrate that machine unlearning can be subverted to implant backdoors covertly, rather than solely serving as a privacy-preserving tool.
• To develop detection and mitigation methods for unlearning-based backdoor attacks.
• To evaluate the effectiveness of the proposed attack under both exact and approximate unlearning (SISA) settings.
• To analyze the impact of data sharding on the detectability and success of the attack and defense mechanisms.

Proposed method

• The attacker first trains a benign model on a dataset augmented with poison samples (labeled with a target class) and mitigation samples (labeled with the correct class).
• The mitigation samples are carefully designed to be indistinguishable from clean data but are used as targets for unlearning requests.
• The attacker submits multiple unlearning requests for the mitigation samples, causing the model to gradually forget their influence and reconfigure decision boundaries to favor the backdoor.
• The backdoor is activated during the unlearning process, as the model's parameters shift to misclassify inputs containing the trigger.
• Two defense mechanisms are proposed: one based on identifying suspicious unlearning requests by analyzing the impact of removed samples, and another using influence functions to detect mitigation samples.
• The attack is evaluated in both exact unlearning and SISA (approximate unlearning) settings, with SISA using data sharding and slice-based retraining.

Experimental results

Research questions

• RQ1 Can machine unlearning be exploited to covertly implant a backdoor in a deep learning model without initial detection?
• RQ2 How effective is the proposed BAU attack in increasing the backdoor success rate while maintaining low initial attack success rates?
• RQ3 To what extent does data sharding in unlearning systems like SISA reduce the effectiveness of detection mechanisms for unlearning-based backdoors?
• RQ4 Can influence-based or sample-impact analysis effectively detect malicious unlearning requests targeting mitigation samples?
• RQ5 How does the proposed defense mechanism compare to existing backdoor detection techniques in the context of unlearning-based attacks?

Key findings

• The BAU attack successfully implants a backdoor into the model through unlearning requests, with the attack success rate increasing significantly after unlearning mitigation samples.
• The initial model exhibits a low attack success rate, making it difficult for existing backdoor detection methods to identify the malicious behavior before unlearning.
• Data sharding in the SISA unlearning framework increases the computational overhead and reduces the effectiveness of the proposed detection algorithms.
• The detection method based on identifying mitigation samples through influence analysis is effective in identifying malicious unlearning requests in non-sharded settings.
• The defense mechanism based on influence functions successfully identifies the mitigation samples used in the attack, demonstrating feasibility in detecting unlearning-based backdoor propagation.
• The attack remains effective in both exact unlearning and approximate unlearning (SISA) settings, highlighting the vulnerability of current unlearning frameworks to adversarial manipulation.