[Paper Review] When Not to Classify: Anomaly Detection of Attacks (ADA) on DNN Classifiers at Test Time

David J. Miller, Yulia Wang+1 more|arXiv (Cornell University)|Dec 18, 2017

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes an unsupervised anomaly detection (AD) system for detecting test-time evasion attacks on deep neural network (DNN) classifiers by modeling the joint density of DNN penultimate layers using mixture distributions matched to ReLU activation support. It introduces a novel Kullback-Leibler divergence-based decision statistic leveraging source/destination class concepts and weight information, achieving state-of-the-art AUC performance on MNIST and CIFAR-10 under multiple attack types, including the strong Carlini-Wagner (CW) attack.

ABSTRACT

A significant threat to the recent, wide deployment of machine learning-based systems, including deep neural networks (DNNs), is adversarial learning attacks. We analyze possible test-time evasion-attack mechanisms and show that, in some important cases, when the image has been attacked, correctly classifying it has no utility: i) when the image to be attacked is (even arbitrarily) selected from the attacker's cache; ii) when the sole recipient of the classifier's decision is the attacker. Moreover, in some application domains and scenarios it is highly actionable to detect the attack irrespective of correctly classifying in the face of it (with classification still performed if no attack is detected). We hypothesize that, even if human-imperceptible, adversarial perturbations are machine-detectable. We propose a purely unsupervised anomaly detector (AD) that, unlike previous works: i) models the joint density of a deep layer using highly suitable null hypothesis density models (matched in particular to the non- negative support for RELU layers); ii) exploits multiple DNN layers; iii) leverages a "source" and "destination" class concept, source class uncertainty, the class confusion matrix, and DNN weight information in constructing a novel decision statistic grounded in the Kullback-Leibler divergence. Tested on MNIST and CIFAR-10 image databases under three prominent attack strategies, our approach outperforms previous detection methods, achieving strong ROC AUC detection accuracy on two attacks and better accuracy than recently reported for a variety of methods on the strongest (CW) attack. We also evaluate a fully white box attack on our system. Finally, we evaluate other important performance measures, such as classification accuracy, versus detection rate and attack strength.

Motivation & Objective

• To address the gap in defense mechanisms against test-time evasion attacks on DNN classifiers, especially in scenarios where correct classification under attack is functionally useless.
• To detect adversarial perturbations as anomalies, even when they are imperceptible to humans, by exploiting statistical deviations in DNN feature representations.
• To develop an unsupervised AD system that does not require labeled attack data or prior knowledge of attack patterns.
• To improve detection robustness by leveraging multiple DNN layers, class confusion matrices, and DNN weight information.
• To provide a detection mechanism that enables safe system responses (e.g., rejection or mitigation) when attacks are detected, even if classification accuracy is not preserved.

Proposed method

• Models the joint density of DNN penultimate layer activations using a mixture of Gaussian and generalized gamma distributions, tailored to the non-negative support of ReLU units.
• Constructs a novel decision statistic based on the Kullback-Leibler (KL) divergence between the empirical distribution of a sample and a null model derived from clean data.
• Incorporates source class uncertainty and class confusion matrix information to refine the anomaly score, improving sensitivity to subtle perturbations.
• Uses DNN weight information to calibrate the null model and enhance the statistical robustness of the anomaly detection process.
• Employs a multi-layer approach by aggregating anomaly scores across multiple DNN layers to improve detection reliability.
• Operates in a purely unsupervised manner, requiring no labeled adversarial examples during training.

Experimental results

Research questions

• RQ1 Can adversarial perturbations—despite being imperceptible to humans—be detected as statistical anomalies in the internal representations of DNNs?
• RQ2 Does modeling the joint density of DNN features using distributions matched to ReLU activation support improve anomaly detection performance?
• RQ3 Can a decision statistic based on KL divergence, enriched with class confusion and weight information, outperform existing unsupervised AD methods on evasion attacks?
• RQ4 How effective is the proposed AD system against strong, state-of-the-art attacks such as the Carlini-Wagner (CW) attack?
• RQ5 Can the system detect attacks even when correct classification of the perturbed input is not useful or actionable?

Key findings

• The proposed AD system achieves strong ROC AUC performance on MNIST and CIFAR-10 under three prominent evasion attack strategies, including the highly robust Carlini-Wagner (CW) attack.
• On the CW attack, the method achieves better detection accuracy than recently reported state-of-the-art methods, demonstrating superior robustness.
• The system maintains high detection rates even under low-strength attacks, with minimal degradation in clean sample classification accuracy.
• The method outperforms prior unsupervised AD approaches by leveraging multi-layer feature representations and a statistically grounded decision statistic.
• A fully white-box attack on the AD system was evaluated, showing that the detector remains resilient under full knowledge of the defense mechanism.
• The approach generalizes well across different DNN architectures and is applicable to non-image domains such as speech, genomics, and IoT sensor data with appropriate null model adaptation.