[Paper Review] Detecting Backdoor Poisoning Attacks on Deep Neural Networks by Heatmap Clustering

Lukas Schulth, Christian Berghoff+1 more|arXiv (Cornell University)|Apr 27, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes Heatmap Clustering (HC), a novel method for detecting backdoor poisoning attacks in deep neural networks by applying k-means clustering to heatmaps generated via Layer-wise Relevance Propagation (LRP). HC outperforms Activation Clustering (AC) in detecting standard and label-consistent backdoor attacks, especially under reduced trigger visibility, achieving up to 99.82% accuracy and 100% true positive rate on 33% poisoned data, with significantly lower variance in performance.

ABSTRACT

Predicitions made by neural networks can be fraudulently altered by so-called poisoning attacks. A special case are backdoor poisoning attacks. We study suitable detection methods and introduce a new method called Heatmap Clustering. There, we apply a $k$-means clustering algorithm on heatmaps produced by the state-of-the-art explainable AI method Layer-wise relevance propagation. The goal is to separate poisoned from un-poisoned data in the dataset. We compare this method with a similar method, called Activation Clustering, which also uses $k$-means clustering but applies it on the activation of certain hidden layers of the neural network as input. We test the performance of both approaches for standard backdoor poisoning attacks, label-consistent poisoning attacks and label-consistent poisoning attacks with reduced amplitude stickers. We show that Heatmap Clustering consistently performs better than Activation Clustering. However, when considering label-consistent poisoning attacks, the latter method also yields good detection performance.

Motivation & Objective

• Address the challenge of detecting backdoor poisoning attacks, particularly label-consistent variants that evade human inspection due to consistent labeling.
• Develop a robust detection method that identifies poisoned data points without relying on human-perceptible trigger patterns.
• Evaluate the effectiveness of heatmap-based clustering against activation-based clustering in detecting various backdoor attack variants.
• Demonstrate that label-consistent backdoor attacks—previously thought undetectable—are detectable using explainable AI techniques like LRP.

Proposed method

• Apply Layer-wise Relevance Propagation (LRP) to generate input-activation heatmaps for each data point, visualizing feature importance in the network.
• Use k-means clustering on the LRP heatmaps to group data points based on their decision-making strategies, with Gromov-Wasserstein distance as the similarity measure.
• Compare the clustering results to identify outliers corresponding to poisoned samples, assuming poisoned data follows a distinct decision pattern.
• Implement a baseline method, Activation Clustering (AC), which applies k-means to activations from intermediate layers instead of heatmaps.
• Use the Gromov-Wasserstein distance to measure structural similarity between heatmaps, enabling comparison of complex, non-Euclidean relevance distributions.
• Evaluate detection performance using metrics like detection accuracy (ACC), true positive rate (TPR), and true negative rate (TNR) across multiple attack configurations.

Experimental results

Research questions

• RQ1 Can heatmap clustering via LRP detect standard backdoor poisoning attacks more effectively than activation clustering?
• RQ2 Does heatmap clustering remain effective when triggers are made visually imperceptible through amplitude reduction?
• RQ3 Can label-consistent backdoor attacks—where poisoned samples are labeled correctly—still be detected using explainable AI methods?
• RQ4 How does the robustness of heatmap clustering compare to activation clustering under variations in trigger position and image selection?
• RQ5 Is there a performance trade-off between detection accuracy and computational cost in heatmap clustering?

Key findings

• Heatmap Clustering (HC) achieved a mean detection accuracy of 99.82% and a true positive rate of 99.46% on 33% poisoned data in label-consistent attacks, significantly outperforming Activation Clustering.
• For attacks with reduced trigger amplitude (e.g., 32), HC maintained a detection accuracy of 81.58% and TPR of 58.46%, while AC dropped to 78.97% accuracy and 74.26% TPR.
• HC showed near-zero variance in detection performance (standard deviation of 0.19 for accuracy), whereas AC had a standard deviation of 6.81, indicating superior robustness.
• In experiments with 33% poisoned data and reduced trigger amplitude, the attack success rate (ASR) dropped to 25.65% for AC and 81.58% for HC, showing HC's resilience to subtle triggers.
• The study refutes the claim that label-consistent backdoor attacks are undetectable, demonstrating that such attacks can be identified using LRP-based heatmap analysis.
• HC has a significantly longer runtime than AC due to the computationally intensive Gromov-Wasserstein distance calculation, suggesting a hybrid detection pipeline may be optimal.