[Paper Review] Detecting Adversarial Samples from Artifacts

Reuben Feinman, Ryan R. Curtin+2 more|arXiv (Cornell University)|Mar 1, 2017

Adversarial Robustness in Machine LearningComputer Science17 references376 citations

TL;DR

Proposes a two-feature detector for adversarial samples using density estimation in last-layer feature space and Bayesian uncertainty from dropout; combines them in logistic regression to detect adversarial inputs across multiple attacks and datasets.

ABSTRACT

Deep neural networks (DNNs) are powerful nonlinear architectures that are known to be robust to random perturbations of the input. However, these models are vulnerable to adversarial perturbations--small input changes crafted explicitly to fool the model. In this paper, we ask whether a DNN can distinguish adversarial samples from their normal and noisy counterparts. We investigate model confidence on adversarial samples by looking at Bayesian uncertainty estimates, available in dropout neural networks, and by performing density estimation in the subspace of deep features learned by the model. The result is a method for implicit adversarial detection that is oblivious to the attack algorithm. We evaluate this method on a variety of standard datasets including MNIST and CIFAR-10 and show that it generalizes well across different architectures and attacks. Our findings report that 85-93% ROC-AUC can be achieved on a number of standard classification tasks with a negative class that consists of both normal and noisy samples.

Motivation & Objective

• Motivate the need to detect adversarial samples beyond attack-specific defenses.
• Develop two attack-agnostic features that reveal artifacts of adversarial perturbations.
• Demonstrate a simple classifier that combines density and uncertainty to detect adversarial inputs.
• Evaluate generalization across multiple datasets, architectures, and attacks.
• Analyze how adversarial perturbations move inputs off the data manifold and into high-uncertainty regions.

Proposed method

• Use kernel density estimation in the last hidden layer feature space to measure how far a sample lies from the class manifolds.
• Leverage dropout-based Bayesian uncertainty estimates from Bayesian neural networks to identify low-confidence regions.
• Compute a two-feature input (uncertainty, density) and train a logistic regression detector to distinguish adversarial from normal/noisy samples.
• Evaluate multiple attacks (FGSM, BIM, JSMA, C&W) and datasets (MNIST, CIFAR-10, SVHN) through ROC-AUC metrics.
• Fix that density bandwidth via likelihood-based selection and apply z-score normalization to features before training the detector.

Experimental results

Research questions

• RQ1 Can adversarial samples be distinguished from normal and noisy samples using manifold-based density estimates in the last-layer feature space?
• RQ2 Can Bayesian neural network uncertainty, obtained via dropout, reveal adversarial inputs not captured by density alone?
• RQ3 How well does a simple logistic regression detector perform when combining density and uncertainty features across diverse attacks and datasets?
• RQ4 Do the proposed features generalize across architectures and attack types without attack-specific training?
• RQ5 What is the detectable performance (ROC-AUC) for adversarial detection on MNIST, CIFAR-10, and SVHN?

Key findings

• A combined detector using density estimates and Bayesian uncertainty achieves high ROC-AUC across attacks and datasets (e.g., up to 92.6% on MNIST when negative class includes normal and noisy samples).
• Density estimates tend to decrease and uncertainty tends to increase for adversarial samples, relative to their normal counterparts.
• The combined uncertainty+density detector outperforms either feature alone, particularly for JSMA, BIM-A, and C&W attacks.
• Experiments on MNIST, CIFAR-10, and SVHN show strong ROC-AUC performance across FGSM, BIM, JSMA, and C&W attacks.
• Adversarial detection remains effective when the negative class includes both normal and noisy samples, indicating robustness to perturbation-type variation.