[Paper Review] Demon in the Variant: Statistical Analysis of DNNs for Robust Backdoor Contamination Detection

Di Tang, Xiaofeng Wang+2 more|arXiv (Cornell University)|Aug 2, 2019

Adversarial Robustness in Machine Learning41 references44 citations

TL;DR

Introduces TaCT and SCAn to detect backdoor contamination in DNNs, including source-specific backdoors, by global representation statistics using EM decomposition and likelihood-ratio testing.

ABSTRACT

A security threat to deep neural networks (DNN) is backdoor contamination, in which an adversary poisons the training data of a target model to inject a Trojan so that images carrying a specific trigger will always be classified into a specific label. Prior research on this problem assumes the dominance of the trigger in an image's representation, which causes any image with the trigger to be recognized as a member in the target class. Such a trigger also exhibits unique features in the representation space and can therefore be easily separated from legitimate images. Our research, however, shows that simple target contamination can cause the representation of an attack image to be less distinguishable from that of legitimate ones, thereby evading existing defenses against the backdoor infection. In our research, we show that such a contamination attack actually subtly changes the representation distribution for the target class, which can be captured by a statistic analysis. More specifically, we leverage an EM algorithm to decompose an image into its identity part (e.g., person, traffic sign) and variation part within a class (e.g., lighting, poses). Then we analyze the distribution in each class, identifying those more likely to be characterized by a mixture model resulted from adding attack samples to the legitimate image pool. Our research shows that this new technique effectively detects data contamination attacks, including the new one we propose, and is also robust against the evasion attempts made by a knowledgeable adversary.

Motivation & Objective

• Explain why existing backdoor defenses fail against targeted contamination (TaCT).
• Develop a robust detector (SCAn) that leverages global representation distributions across all classes.
• Demonstrate TaCT can create source-specific backdoors with indistinguishable attack and benign representations.
• Provide empirical evaluation across multiple datasets showing TaCT bypasses prior defenses and SCAn detects contamination.

Proposed method

• Represent inputs as two components via EM: identity (mu_t) and variation (epsilon).
• Decompose representations across all classes to estimate class-specific identity vectors and a universal variation distribution.
• Apply likelihood-ratio tests to detect classes whose representations reflect a contaminated mixture.
• Demonstrate TaCT by inserting trigger plus cover images to create source-specific backdoors with limited contamination.
• Evaluate defenses (Neural Cleanse, STRIP, SentiNet, Activation Clustering) against TaCT and show their failures.
• Propose SCAn as a global-information-based detector leveraging distributions across classes.

Experimental results

Research questions

• RQ1 Can TaCT enable source-specific backdoors that evade existing trigger-dominance defenses?
• RQ2 Does a global representation analysis across all classes reveal contamination not detectable by class-internal methods?
• RQ3 Is EM-based decomposition of representations into identity and variation effective for detecting backdoors?
• RQ4 How robust is SCAn to different backdoor configurations and black-box attacks?
• RQ5 What is the comparative effectiveness of SCAn versus established defenses on diverse datasets?

Key findings

• TaCT enables source-specific backdoors with high targeted misclassification and low disruption to non-source classes using cover images.
• Representations of attack images under TaCT become indistinguishable from normal target-class images in two-dimensional PCA projections.
• Existing defenses (Neural Cleanse, STRIP, SentiNet, Activation Clustering) fail to reliably detect TaCT infections on GTSRB and CIFAR-10.
• TaCT achieves high targeted misclassification rates with modest contamination (e.g., 2.1% contamination including cover images) while keeping overall accuracy near baseline.
• SCAn leverages two-component decomposition and universal variation to detect contamination by analyzing cross-class distribution inconsistencies.
• SCAn demonstrates effectiveness against TaCT and robustness to other black-box attacks by exploiting global information across classes.