[Paper Review] Improving Entropic Out-of-Distribution Detection using Isometric Distances and the Minimum Distance Score.

David Macêdo, Teresa B. Ludermir|arXiv (Cornell University)|May 30, 2021

Anomaly Detection Techniques and Applications30 references4 citations

TL;DR

This paper improves entropic out-of-distribution detection by isometrizing distances in the IsoMax loss and replacing the entropic score with a minimum distance score. These modifications enhance detection performance without altering model architecture or training procedures, maintaining the seamless nature of the original approach while achieving superior OOD detection accuracy.

ABSTRACT

Current out-of-distribution detection approaches usually present special requirements (e.g., collecting outlier data and hyperparameter validation) and produce side effects (classification accuracy drop and slow/inefficient inferences). Recently, entropic out-of-distribution detection has been proposed as a seamless approach (i.e., a solution that avoids all the previously mentioned drawbacks). The entropic out-of-distribution detection solution comprises the IsoMax loss for training and the entropic score for out-of-distribution detection. The IsoMax loss works as a SoftMax loss drop-in replacement because swapping the SoftMax loss with the IsoMax loss requires no changes in the model's architecture or training procedures/hyperparameters. In this paper, we propose to perform what we call an isometrization of the distances used in the IsoMax loss. Additionally, we propose to replace the entropic score with the minimum distance score. Our experiments showed that these simple modifications increase out-of-distribution detection performance while keeping the solution seamless. Code available at $\href{this https URL}{ ext{entropic out-of-distribution detection}}$.

Motivation & Objective

• To address the limitations of existing out-of-distribution detection methods that require outlier data collection and hyperparameter tuning.
• To improve the performance of entropic out-of-distribution detection, which is already seamless but can be further optimized.
• To maintain the seamless nature of the original method by avoiding changes to model architecture or training procedures.
• To explore whether isometric distance regularization and a minimum distance score can enhance detection robustness and accuracy.

Proposed method

• Isometrization of distances in the IsoMax loss to preserve relative inter-class distances during training.
• Replacement of the entropic score with the minimum distance score for out-of-distribution detection.
• Use of the IsoMax loss as a direct SoftMax replacement without modifying model architecture or training hyperparameters.
• Application of the minimum distance score to measure the smallest distance from a sample to any known class center in the feature space.
• Preservation of the original method's seamless nature by avoiding additional data collection or inference-time adjustments.
• Integration of isometric distance regularization and minimum distance scoring into the existing entropic OOD detection framework.

Experimental results

Research questions

• RQ1 Can isometric distance regularization in the IsoMax loss improve out-of-distribution detection performance?
• RQ2 Does replacing the entropic score with the minimum distance score lead to better OOD detection without compromising the method's seamless nature?
• RQ3 How do these modifications affect detection accuracy and inference efficiency compared to the original entropic approach?
• RQ4 To what extent can performance be improved while maintaining compatibility with standard training pipelines?

Key findings

• The proposed isometrization of distances in the IsoMax loss leads to improved feature space geometry, enhancing OOD detection performance.
• Replacing the entropic score with the minimum distance score results in stronger discrimination between in-distribution and out-of-distribution samples.
• The method maintains full compatibility with existing training pipelines, requiring no changes to model architecture or hyperparameters.
• The combination of isometric distances and the minimum distance score achieves superior OOD detection performance compared to the original entropic approach.
• The improvements are achieved without introducing side effects such as accuracy drops or inference inefficiencies.
• The solution remains seamless, preserving the key advantage of the original entropic OOD detection method.