[Paper Review] Deep Anomaly Detection with Outlier Exposure

Dan Hendrycks, Mantas Mazeika+1 more|arXiv (Cornell University)|Dec 11, 2018

Anomaly Detection Techniques and Applications401 citations

TL;DR

Outlier Exposure (OE) trains anomaly detectors using an auxiliary outlier dataset to improve detection of unseen anomalies across vision and NLP tasks, improving OOD detection and calibration.

ABSTRACT

It is important to detect anomalous inputs when deploying machine learning systems. The use of larger and more complex inputs in deep learning magnifies the difficulty of distinguishing between anomalous and in-distribution examples. At the same time, diverse image and text data are available in enormous quantities. We propose leveraging these data to improve deep anomaly detection by training anomaly detectors against an auxiliary dataset of outliers, an approach we call Outlier Exposure (OE). This enables anomaly detectors to generalize and detect unseen anomalies. In extensive experiments on natural language processing and small- and large-scale vision tasks, we find that Outlier Exposure significantly improves detection performance. We also observe that cutting-edge generative models trained on CIFAR-10 may assign higher likelihoods to SVHN images than to CIFAR-10 images; we use OE to mitigate this issue. We also analyze the flexibility and robustness of Outlier Exposure, and identify characteristics of the auxiliary dataset that improve performance.

Motivation & Objective

• Motivate robust anomaly detection for deep learning systems under distribution shift.
• Propose Outlier Exposure (OE) by training with an auxiliary outlier dataset to learn generalizable OOD cues.
• ShowOE enhances existing OOD detectors across tasks and architectures.
• Demonstrate OE benefits for density estimation and classifier calibration under OOD exposure.

Proposed method

• Formalize OE as augmenting the training objective with a term encouraging reduced confidence on OE outliers.
• Fine-tune or train networks so that their outputs on OE data are pushed toward a uniform distribution (or lower confidence) to discourage overconfidence on unseen anomalies.
• Use diverse, real-world OE datasets (e.g., 80 Million Tiny Images, ImageNet-22K, WikiText-2) as outliers rather than synthetic noise or GANs.
• Apply OE to multiclass classifiers in vision and NLP, using MSP and a confidence-branch baseline as OOD detectors.
• Explore OE with density estimators (PixelCNN++) and language models (QRNN) to improve OOD detection via adjusted likelihood signals.
• Investigate dataset diversity and closeness effects, and assess calibration improvements under realistic OOD test settings.

Experimental results

Research questions

• RQ1 Can Outlier Exposure improve the generalization of OOD detectors to unseen anomaly distributions?
• RQ2 Does incorporating real, diverse outlier data outperform synthetic or noise-based outliers for OOD detection?
• RQ3 How does OE interact with existing detectors (MSP, confidence branch) and with density estimation and language models?
• RQ4 What is OE’s impact on calibration of predictions when some data are OOD?
• RQ5 What characteristics of OE datasets (diversity, closeness) influence performance?

Key findings

• OE consistently improves average OOD detection performance across vision and NLP tasks.
• Real, diverse OE data outperforms synthetic or GAN-generated outliers for most setups.
• OE enhances density-estimation based OOD scoring toward more plausible densities for outliers.
• OE improves calibration of neural classifiers in realistic settings with mixed in-distribution and out-of-distribution data.
• Using OE with existing detectors yields substantial gains without tuning to specific test-out distributions.