[Paper Review] On Detecting Adversarial Perturbations

Jan Hendrik Metzen, Tim Genewein+2 more|arXiv (Cornell University)|Feb 14, 2017

Adversarial Robustness in Machine Learning219 citations

TL;DR

The paper adds a small detector subnet to a classifier to distinguish genuine data from adversarial examples, showing strong detectability on CIFAR-10 and a 10-class ImageNet subset, including defenses against dynamic adversaries.

ABSTRACT

Machine learning and deep learning in particular has advanced tremendously on perceptual tasks in recent years. However, it remains vulnerable against adversarial perturbations of the input that have been crafted specifically to fool the system while being quasi-imperceptible to a human. In this work, we propose to augment deep neural networks with a small "detector" subnetwork which is trained on the binary classification task of distinguishing genuine data from data containing adversarial perturbations. Our method is orthogonal to prior work on addressing adversarial perturbations, which has mostly focused on making the classification network itself more robust. We show empirically that adversarial perturbations can be detected surprisingly well even though they are quasi-imperceptible to humans. Moreover, while the detectors have been trained to detect only a specific adversary, they generalize to similar and weaker adversaries. In addition, we propose an adversarial attack that fools both the classifier and the detector and a novel training procedure for the detector that counteracts this attack.

Motivation & Objective

• Motivate and address the vulnerability of deep networks to quasi-imperceptible adversarial perturbations.
• Propose a binary detector subnet that differentiates original data from adversarially perturbed data.
• Show detector generalization across similar and weaker adversaries beyond the training adversary.
• Investigate dynamic adversaries and propose training strategies to harden detectors against them.

Proposed method

• Attach a small adversary detector subnet to a pre-trained classifier at an intermediate layer.
• Train the detector on a balanced dataset of original vs. adversarial examples generated for the training set.
• Freeze classifier weights and train the detector with cross-entropy loss for adversary labels.
• Probe detector placement and architecture through experiments on CIFAR-10 and ImageNet subset.
• Introduce a dynamic adversary formulation that optimizes both classifier and detector objectives during perturbation generation.
• Develop dynamic adversary training to harden detectors against adaptive attacks.

Experimental results

Research questions

• RQ1 Can data-dependent adversarial perturbations be reliably detected by a detector trained on a specific adversary?
• RQ2 How does detector placement within the classifier affect adversarial detectability?
• RQ3 Do detectors trained on one adversary transfer to other adversaries or norms (e.g., l_inf vs l2)?
• RQ4 How robust are detectors to dynamic adversaries that adapt to both classifier and detector?
• RQ5 What training procedure can harden detectors against adaptive, dynamic attacks?

Key findings

• Detectors achieve high detectability (above 80%) across tested adversaries on CIFAR-10, with higher detectability (>90%) when classifier accuracy on adversarial examples falls below 10%.
• Detector placement in middle network layers (AD(2)) generally yields best detection for fast/iterative adversaries; for DeepFool variants, AD(4) is often optimal.
• Detectors trained on one adversary can transfer to other similar/weaker adversaries; transfers between l_inf and l2 variants are often effective for related attacks.
• Dynamic detectors (trained to resist adaptive attacks) maintain detectability >70% across a range of adaptation strengths (sigma values).
• On a 10-class ImageNet subset, detectors attain 85%+ detectability for most adversaries; one iterative l2 case (epsilon=400) approaches chance performance, highlighting a challenging edge case.
• Detectors can enable fallback or safety interventions (e.g., human verification) when adversarial inputs are detected.