[Paper Review] Out-of-Distribution Detection with Deep Nearest Neighbors

Yiyou Sun, Yifei Ming+2 more|arXiv (Cornell University)|Apr 13, 2022

Anomaly Detection Techniques and ApplicationsComputer Science61 citations

TL;DR

This paper proposes a non-parametric KNN-based OOD detector in deep feature space, achieving strong performance without distributional assumptions and scaling to large datasets like ImageNet, often outperforming parametric baselines.

ABSTRACT

Out-of-distribution (OOD) detection is a critical task for deploying machine learning models in the open world. Distance-based methods have demonstrated promise, where testing samples are detected as OOD if they are relatively far away from in-distribution (ID) data. However, prior methods impose a strong distributional assumption of the underlying feature space, which may not always hold. In this paper, we explore the efficacy of non-parametric nearest-neighbor distance for OOD detection, which has been largely overlooked in the literature. Unlike prior works, our method does not impose any distributional assumption, hence providing stronger flexibility and generality. We demonstrate the effectiveness of nearest-neighbor-based OOD detection on several benchmarks and establish superior performance. Under the same model trained on ImageNet-1k, our method substantially reduces the false positive rate (FPR@TPR95) by 24.77% compared to a strong baseline SSD+, which uses a parametric approach Mahalanobis distance in detection. Code is available: https://github.com/deeplearning-wisc/knn-ood.

Motivation & Objective

• Motivate and evaluate a non-parametric approach to OOD detection that avoids distributional assumptions about the feature space.
• Show that deep KNN distance can separate ID and OOD data using a simple threshold on k-NN distances.
• Demonstrate robustness across model architectures and training losses (CE and SupCon).
• Provide theoretical justification linking KNN-based decisions to Bayes-optimal detection under a contamination model.

Proposed method

• Compute normalized penultimate-layer features z = phi(x)/||phi(x)||_2 for both training and test samples.
• At test time, compute the k-th nearest-neighbor distance r_k(z*) to training embeddings Z_n.
• Declare OOD if -r_k(z*) >= lambda, where lambda is chosen so a high fraction (e.g., 95%) of ID data is correctly classified.
• Use Faiss for efficient approximate nearest-neighbor search over precomputed ID embeddings.
• Show compatibility with different backbones (CNNs and ViT) and training losses (CE and SupCon).
• Optionally apply activation rectification (ReAct) and compare k-th vs. averaged k distances.

Experimental results

Research questions

• RQ1 Can a non-parametric nearest-neighbor distance suffice for effective OOD detection without assuming a parametric form of the feature space?
• RQ2 How does KNN-based OOD detection perform across different architectures (CNNs, ViTs) and training losses (CE, SupCon)?
• RQ3 What design choices (normalization, feature space selection, k value, sampling) most affect OOD detection performance and speed?
• RQ4 Does KNN-based OOD detection scale to large datasets like ImageNet with competitive inference time?

Key findings

• KNN-based OOD detection substantially reduces false positives compared to a strong Mahalanobis baseline under the same ImageNet-1k model (FPR@TPR95 improved by 24.77%).
• On CIFAR-10 with SupCon, KNN+ achieves lower FPR95 than SSD+ across hard OOD tasks, indicating non-parametric methods’ advantage with contrastive representations.
• Table-cited results show KNN+ attains 11.07 FPR95 vs IForest, OCSVM, LOF, etc., on CIFAR-10 averaged across datasets, indicating strong non-parametric performance.
• KNN-based methods maintain competitive inference speed with large datasets (ImageNet-scale) and can match full-dataset performance with only 1% sampling.
• Across ViT-B/16, KNN outperforms Mahalanobis on multiple OOD datasets (iNaturalist, SUN, Places, Textures).
• Activation rectification (ReAct) further boosts KNN+ performance (FPR95 reduction from 38.47 to 26.45 on ImageNet-1k).