[Paper Review] Energy-based Out-of-distribution Detection

Weitang Liu, Xiaoyun Wang+2 more|arXiv (Cornell University)|Oct 8, 2020

Adversarial Robustness in Machine Learning48 references430 citations

TL;DR

The paper introduces an energy score for OOD detection that outperforms softmax-based methods, and shows how to use energy as a scoring function at inference or as a trainable objective to shape the energy landscape for better separation of in-distribution and OOD data.

ABSTRACT

Determining whether inputs are out-of-distribution (OOD) is an essential building block for safely deploying machine learning models in the open world. However, previous methods relying on the softmax confidence score suffer from overconfident posterior distributions for OOD data. We propose a unified framework for OOD detection that uses an energy score. We show that energy scores better distinguish in- and out-of-distribution samples than the traditional approach using the softmax scores. Unlike softmax confidence scores, energy scores are theoretically aligned with the probability density of the inputs and are less susceptible to the overconfidence issue. Within this framework, energy can be flexibly used as a scoring function for any pre-trained neural classifier as well as a trainable cost function to shape the energy surface explicitly for OOD detection. On a CIFAR-10 pre-trained WideResNet, using the energy score reduces the average FPR (at TPR 95%) by 18.03% compared to the softmax confidence score. With energy-based training, our method outperforms the state-of-the-art on common benchmarks.

Motivation & Objective

• Motivate the need for reliable OOD detection in open-world deployment.
• Propose an energy-based framework that correlates with input density to distinguish in-distribution from OOD data.
• Show how to use the energy score with pre-trained classifiers without retraining and how to train models to increase the energy gap between in-distribution and OOD data.
• Demonstrate improved OOD detection performance on CIFAR-10/ CIFAR-100 with WideResNet compared to softmax-based scores.
• Provide practical guidance on parameter choices and compare with existing methods including Outlier Exposure and JEM-based approaches.

Proposed method

• Define an energy function E(x;f) = -T log sum_i e^{f_i(x)/T} derived from a discriminative classifier f.
• Use -E(x;f) as a scoring function to detect OOD with a threshold on the energy.
• Show that log softmax confidence is a special case of the free energy, explaining why energy can outperform softmax for OOD detection.
• Offer an energy-bounded learning objective that jointly minimizes standard cross-entropy and a hinge-based energy regularizer to create a gap between in-distribution and OOD energies.
• Fine-tune with unlabeled OOD data using two hinge terms with margins m_in and m_out to push in-distribution energies below m_in and OOD energies above m_out.
• Demonstrate that energy-based scoring is parameter-free at inference (T=1) and compare inference-time usage with and without fine-tuning.

Experimental results

Research questions

• RQ1 Can an energy-based score provide a theoretically grounded and practically superior alternative to softmax confidence for OOD detection?
• RQ2 How does the energy score compare to softmax scores on standard OOD benchmarks when using pre-trained models without retraining?
• RQ3 Does energy-based fine-tuning with outlier data improve OOD detection without sacrificing in-distribution accuracy?
• RQ4 What are the effects of temperature scaling and energy-margin parameters on OOD detection performance?
• RQ5 How does energy-based OOD detection compare with generative-model-based and hybrid approaches in terms of performance and training stability?

Key findings

• Energy scores outperform softmax confidence for OOD detection on CIFAR-10 with WideResNet, reducing FPR95 by about 18.03% on average without fine-tuning.
• Energy-based fine-tuning with unlabeled outlier data further improves OOD metrics, reducing FPR95 by up to 10.55% overOutlier Exposure on CIFAR-10/ CIFAR-100 while preserving accuracy.
• Negative energy scores provide more distinguishable separation between in-distribution and OOD data than softmax scores, as illustrated by smoother energy distributions.
• Energy-based methods achieve competitive or superior AUROC and AUPR across several OOD benchmarks (iSUN, Places365, Texture, SVHN, LSUN variants) compared to baselines, including discriminative OOD detectors and some generative-model-based methods.
• Temperature scaling (T>1) degrades energy-based OOD detection, indicating T=1 is preferable for this approach.