[Paper Review] A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks

Kimin Lee, Kibok Lee+2 more|arXiv (Cornell University)|Jul 10, 2018

Adversarial Robustness in Machine Learning976 citations

TL;DR

The paper introduces a Mahalanobis distance-based confidence score in DNN feature space to detect OOD and adversarial samples using a pre-trained softmax classifier, with input pre-processing and feature ensemble enhancements, and demonstrates robustness and applicability to class-incremental learning.

ABSTRACT

Detecting test samples drawn sufficiently far away from the training distribution statistically or adversarially is a fundamental requirement for deploying a good classifier in many real-world machine learning applications. However, deep neural networks with the softmax classifier are known to produce highly overconfident posterior distributions even for such abnormal samples. In this paper, we propose a simple yet effective method for detecting any abnormal samples, which is applicable to any pre-trained softmax neural classifier. We obtain the class conditional Gaussian distributions with respect to (low- and upper-level) features of the deep models under Gaussian discriminant analysis, which result in a confidence score based on the Mahalanobis distance. While most prior methods have been evaluated for detecting either out-of-distribution or adversarial samples, but not both, the proposed method achieves the state-of-the-art performances for both cases in our experiments. Moreover, we found that our proposed method is more robust in harsh cases, e.g., when the training dataset has noisy labels or small number of samples. Finally, we show that the proposed method enjoys broader usage by applying it to class-incremental learning: whenever out-of-distribution samples are detected, our classification rule can incorporate new classes well without further training deep models.

Motivation & Objective

• Motivate reliable detection of abnormal test samples drawn far from training distribution or adversarially perturbed.
• Propose a simple generative classifier in the DNN feature space under Gaussian discriminant analysis.
• Enable detection without retraining of pre-trained softmax classifiers.
• Improve robustness to noisy labels and small training samples.
• Demonstrate applicability to class-incremental learning by updating class means and shared covariance.

Proposed method

• Fit class-conditional Gaussians with a tied covariance to DNN penultimate-layer features using empirical means and pooled covariance.
• Define a Mahalanobis distance-based confidence score M(x) = max_c -(f(x)-mu_c)^T Sigma^{-1}(f(x)-mu_c).
• Justify that the generative classifier under GDA aligns with the softmax classifier, preserving accuracy.
• Enhance performance with input pre-processing that perturbs x along the gradient of M(x).
• Enhance robustness with feature ensemble by computing M(x) at multiple network layers and learning weights via logistic regression.
• Apply a simple update rule to accommodate new classes in incremental learning by updating class means and the shared covariance.

Experimental results

Research questions

• RQ1 Can a Mahalanobis-distance-based score in DNN feature space outperform softmax-based confidence for OOD and adversarial detection?
• RQ2 Does combining multi-layer features and input pre-processing improve detection robustness under noise and limited data?
• RQ3 Can the same framework support class-incremental learning without retraining the entire model?
• RQ4 Is the approach robust when hyperparameters are tuned using in-distribution or adversarial data only?
• RQ5 How does the method perform across diverse datasets and architectures (e.g., CIFAR-10/100, SVHN, ImageNet, LSUN)?

Key findings

• The Mahalanobis-based score outperforms softmax-based baselines and competing detectors (e.g., ODIN, LID) for both OOD and adversarial detection across multiple datasets.
• Input pre-processing and feature ensemble significantly boost detection performance, including high TNR at 95% TPR and strong AUROC.
• The detector remains robust under noisy labels and small training sets, and can be tuned using in-distribution or FGSM-adversarial data alone.
• The approach supports class-incremental learning by updating class means and the shared covariance to accommodate new classes without retraining deep models.
• The method achieves state-of-the-art results for OOD detection on several pairs (e.g., CIFAR-10 vs. LSUN/TinyImageNet) and for adversarial attacks (FGSM, BIM, DeepFool, CW).