[Paper Review] Defense against Backdoor Attacks via Identifying and Purifying Bad Neurons

Mingyuan Fan, Yang Liu+3 more|arXiv (Cornell University)|Aug 13, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes WIPER, a novel backdoor defense method that identifies and purifies bad neurons using a new metric, benign salience (BS), which leverages first-order gradients to better capture inter-neuron connections. It further introduces adaptive regularization (AR) for fine-tuning, enabling faster and more stable purification than standard methods, achieving over 90% ASR reduction with less than 3.5% accuracy drop even at 1% clean data ratio.

ABSTRACT

The opacity of neural networks leads their vulnerability to backdoor attacks, where hidden attention of infected neurons is triggered to override normal predictions to the attacker-chosen ones. In this paper, we propose a novel backdoor defense method to mark and purify the infected neurons in the backdoored neural networks. Specifically, we first define a new metric, called benign salience. By combining the first-order gradient to retain the connections between neurons, benign salience can identify the infected neurons with higher accuracy than the commonly used metric in backdoor defense. Then, a new Adaptive Regularization (AR) mechanism is proposed to assist in purifying these identified infected neurons via fine-tuning. Due to the ability to adapt to different magnitudes of parameters, AR can provide faster and more stable convergence than the common regularization mechanism in neuron purifying. Extensive experimental results demonstrate that our method can erase the backdoor in neural networks with negligible performance degradation.

Motivation & Objective

• To address the limitations of existing backdoor defenses that rely on activation magnitude (AM) for neuron evaluation, which often misidentify good neurons as bad due to ignoring inter-neuron connections.
• To overcome performance degradation in existing defenses caused by indiscriminate pruning of neurons, especially those involved in normal prediction pathways.
• To develop a more accurate and stable method for identifying and purifying bad neurons that preserves model accuracy while effectively erasing backdoor triggers.
• To introduce a novel regularization mechanism that adapts to parameter magnitude, improving convergence and effectiveness during neuron purification.

Proposed method

• Introduce benign salience (BS), a new metric that combines first-order gradients with neuron activation to evaluate neuron importance, improving detection of backdoor-infected neurons over standard activation magnitude (AM).
• Use BS to identify 'bad neurons'—those with high attention to triggers but low relevance to normal features—by preserving gradient-based connectivity information.
• Propose adaptive regularization (AR), a piecewise regularization strategy that dynamically adjusts penalty strength based on parameter magnitude to accelerate and stabilize fine-tuning during purification.
• Apply fine-tuning with AR to purify identified bad neurons instead of direct pruning, reducing performance degradation while effectively suppressing backdoor behavior.
• Integrate BS-based identification and AR-based fine-tuning into a unified defense framework, WIPER, to balance backdoor erasure and model accuracy.

Experimental results

Research questions

• RQ1 Can benign salience (BS) outperform activation magnitude (AM) in identifying backdoor-infected neurons by preserving inter-neuron connectivity?
• RQ2 Does adaptive regularization (AR) improve the convergence speed and stability of neuron purification compared to standard regularization in backdoor defense?
• RQ3 Can fine-tuning with AR-based purification maintain higher model accuracy than pruning-based defenses while achieving superior backdoor erasure?
• RQ4 How does the defense performance of WIPER vary under low-data regimes, such as 1% clean data ratio?

Key findings

• WIPER achieves over 90% ASR reduction with less than 3.5% accuracy drop even when only 1% of the training data is clean, significantly outperforming state-of-the-art methods under low-data conditions.
• Using benign salience (BS) for neuron identification leads to more accurate detection of bad neurons than activation magnitude (AM), reducing the risk of over-pruning good neurons.
• Adaptive regularization (AR) enables faster and more stable convergence during fine-tuning, resulting in more effective suppression of backdoor behavior compared to standard regularization.
• Replacing fine-pruning with AR-based purification improves model accuracy by up to 2.8 percentage points (e.g., from 81.85% to 84.67%) while reducing ASR from 16.50% to 4.68% on the TrojanNN attack.
• The method maintains strong performance across different network architectures, with minimal accuracy degradation even when purifying deeper layers, unlike conventional methods that suffer from performance drops in deeper networks.