[Paper Review] Feature Space Singularity for Out-of-Distribution Detection

Haiwen Huang, Zhihan Li+4 more|arXiv (Cornell University)|Nov 30, 2020

Anomaly Detection Techniques and Applications42 references27 citations

TL;DR

This paper proposes Feature Space Singularity Distance (FSSD), a simple yet state-of-the-art method for out-of-distribution (OoD) detection that identifies OoD samples by measuring their distance to a central point—Feature Space Singularity (FSS)—in the neural network's feature space. The method achieves SOTA performance across diverse benchmarks, with a 5% AUROC improvement on large-scale datasets like CelebA and MS-1M, and shows robustness to data corruption and enhanced performance via ensembling.

ABSTRACT

Out-of-Distribution (OoD) detection is important for building safe artificial intelligence systems. However, current OoD detection methods still cannot meet the performance requirements for practical deployment. In this paper, we propose a simple yet effective algorithm based on a novel observation: in a trained neural network, OoD samples with bounded norms well concentrate in the feature space. We call the center of OoD features the Feature Space Singularity (FSS), and denote the distance of a sample feature to FSS as FSSD. Then, OoD samples can be identified by taking a threshold on the FSSD. Our analysis of the phenomenon reveals why our algorithm works. We demonstrate that our algorithm achieves state-of-the-art performance on various OoD detection benchmarks. Besides, FSSD also enjoys robustness to slight corruption in test data and can be further enhanced by ensembling. These make FSSD a promising algorithm to be employed in real world. We release our code at \url{https://github.com/megvii-research/FSSD_OoD_Detection}.

Motivation & Objective

• To address the critical need for reliable OoD detection in real-world AI systems where models may encounter unknown or corrupted inputs.
• To identify a universal structural property in trained neural networks where OoD samples with bounded norms concentrate in feature space.
• To develop a lightweight, inference-time OoD detection method that avoids extra training or complex distance computation to the full training set.
• To demonstrate robustness of the method under data corruption and scalability via ensembling.

Proposed method

• The method identifies a central point in the feature space—called Feature Space Singularity (FSS)—toward which OoD samples with bounded norms concentrate after network training.
• It defines FSSD(x) = ||Fθ(x) − F*|| as the L2 distance from a sample’s feature representation to the FSS, which serves as a score for OoD detection.
• The FSS is computed as the centroid of OoD features from a validation set of known OoD data, requiring no additional training.
• The method leverages the training dynamics of in-distribution and OoD features, showing that FSSD approximates the movement of features during training.
• It is compatible with any pre-trained classifier and can be applied at inference time with minimal computational overhead.
• Performance is further enhanced by ensembling FSSD scores from multiple layers or multiple networks, exploiting diversity in feature representations.

Experimental results

Research questions

• RQ1 Does OoD data from diverse distributions consistently concentrate near a single point in the feature space of a trained neural network?
• RQ2 Can the distance from a sample’s feature to this central point (FSSD) serve as a reliable and generalizable OoD detection score?
• RQ3 How does FSSD compare to existing SOTA methods in terms of AUROC and robustness under data corruption?
• RQ4 Can ensembling FSSD scores from different layers or networks further improve detection performance?
• RQ5 What underlying training dynamics explain the emergence of the FSS phenomenon?

Key findings

• FSSD achieves state-of-the-art performance on all evaluated OoD detection benchmarks, with a 5% AUROC improvement on large-scale datasets such as CelebA and MS-1M.
• The method maintains high robustness to test data corruption, such as ImageNet-C, outperforming uncertainty-based and density-based baselines.
• The performance gap between FSSD and other methods increases with larger training set sizes, indicating scalability and strong generalization.
• Ensembling FSSD scores from multiple layers or networks yields further performance gains, demonstrating the method’s compatibility with ensemble techniques.
• The FSS phenomenon is universal across various architectures (ResNet, ResNeXt, LSTM, LeNet) and datasets (CIFAR10, ImageNet, FashionMNIST, genome data), confirming its broad applicability.
• Theoretical and empirical analysis links the FSS phenomenon to training dynamics and the non-Lipschitz behavior of neural networks, particularly through gradient inner product similarity to training data.