[Paper Review] Adversarial Neuron Pruning Purifies Backdoored Deep Models

Dongxian Wu, Yisen Wang|arXiv (Cornell University)|Oct 27, 2021

Adversarial Robustness in Machine LearningComputer Science50 references27 citations

TL;DR

The paper introduces Adversarial Neuron Pruning (ANP), a data-efficient defense that prunes neurons sensitive to adversarial neuron perturbations to remove backdoors in DNNs, using only a small amount of clean data and no backdoor triggers.

ABSTRACT

As deep neural networks (DNNs) are growing larger, their requirements for computational resources become huge, which makes outsourcing training more popular. Training in a third-party platform, however, may introduce potential risks that a malicious trainer will return backdoored DNNs, which behave normally on clean samples but output targeted misclassifications whenever a trigger appears at the test time. Without any knowledge of the trigger, it is difficult to distinguish or recover benign DNNs from backdoored ones. In this paper, we first identify an unexpected sensitivity of backdoored DNNs, that is, they are much easier to collapse and tend to predict the target label on clean samples when their neurons are adversarially perturbed. Based on these observations, we propose a novel model repairing method, termed Adversarial Neuron Pruning (ANP), which prunes some sensitive neurons to purify the injected backdoor. Experiments show, even with only an extremely small amount of clean data (e.g., 1%), ANP effectively removes the injected backdoor without causing obvious performance degradation.

Motivation & Objective

• Identify a vulnerability in backdoored DNNs where adversarial perturbation of neurons reveals backdoor behaviors even without triggers.
• Propose Adversarial Neuron Pruning (ANP) to prune sensitive neurons and purify models without requiring trigger knowledge or extensive fine-tuning.
• Demonstrate that ANP achieves strong backdoor mitigation with very small clean data (as low as 1%).
• Compare ANP to existing repair methods and establish its data efficiency and robustness across multiple backdoor attacks.

Proposed method

• Formalize neuron-wise perturbations that scale weights and biases by small factors to create adversarial neuron perturbations.
• Show that backdoored models are more susceptible to misclassification under neuron perturbations than benign models.
• Define a pruning mask over all neurons and optimize it with a continuous relaxation of a discrete pruning problem.
• Solve a joint objective balancing clean-data accuracy and robustness to adversarial neuron perturbations using projected gradient descent.
• Adapt the method for Batch Normalization by perturbing scale and shift parameters to preserve normalization effects.
• Output a pruned network f(·; m ⊙ w, b) that removes backdoor behavior without fine-tuning.

Experimental results

Research questions

• RQ1 Can backdoored DNNs be distinguished from benign models by their increased susceptibility to adversarial neuron perturbations?
• RQ2 Can pruning neurons identified through adversarial perturbations effectively remove backdoors while preserving clean-data accuracy?
• RQ3 How does ANP perform across diverse backdoor attacks with very limited clean data?
• RQ4 What are the practical hyperparameters (trade-off alpha, perturbation budget epsilon) and their impact on robustness and accuracy?

Key findings

• Adversarial neuron perturbations induce misclassification that tends to push backdoored models toward their target label even without the trigger.
• ANP can significantly reduce backdoor attack success rate (ASR) with only a small decrease in clean accuracy (ACC).
• Compared to fine-tuning, fine-pruning, and other repairs, ANP achieves stronger robustness across multiple backdoor attacks using only 1% of clean data.
• ANP remains effective across various architectures and can prune neurons using a threshold-based or fraction-based approach with generally favorable ACC and ASR trade-offs.
• Hyperparameters alpha and epsilon show robustness to a wide range, with practical defaults providing strong performance against several attacks.