[Paper Review] Negative Label Guided OOD Detection with Pretrained Vision-Language Models

Xue Jiang, Feng Liu+5 more|arXiv (Cornell University)|Mar 29, 2024

Web Data Mining and Analysis4 citations

TL;DR

This paper proposes NegLabel, a novel post-hoc OOD detection method that leverages vast negative labels from large-scale corpora to enhance out-of-distribution detection in vision-language models. By measuring image affinity to in-distribution and negative labels via a learned score, it achieves state-of-the-art performance on multiple benchmarks, including 94.21% AUROC and 25.40% FPR95 on ImageNet-1k, with strong robustness to domain shifts.

ABSTRACT

Out-of-distribution (OOD) detection aims at identifying samples from unknown classes, playing a crucial role in trustworthy models against errors on unexpected inputs. Extensive research has been dedicated to exploring OOD detection in the vision modality. Vision-language models (VLMs) can leverage both textual and visual information for various multi-modal applications, whereas few OOD detection methods take into account information from the text modality. In this paper, we propose a novel post hoc OOD detection method, called NegLabel, which takes a vast number of negative labels from extensive corpus databases. We design a novel scheme for the OOD score collaborated with negative labels. Theoretical analysis helps to understand the mechanism of negative labels. Extensive experiments demonstrate that our method NegLabel achieves state-of-the-art performance on various OOD detection benchmarks and generalizes well on multiple VLM architectures. Furthermore, our method NegLabel exhibits remarkable robustness against diverse domain shifts. The codes are available at https://github.com/tmlr-group/NegLabel.

Motivation & Objective

• To address the lack of effective OOD detection methods that leverage textual information in vision-language models.
• To improve the generalization and robustness of OOD detection under diverse domain shifts.
• To develop a post-hoc method that enhances detection by incorporating a large set of semantically distant negative labels.
• To design a score function that balances affinities to in-distribution and negative labels for better OOD discrimination.

Proposed method

• Proposes a novel OOD score that is positively correlated with image-text similarity to in-distribution (ID) labels and negatively correlated with similarity to negative labels.
• Introduces the NegMining algorithm to select high-quality negative labels based on their semantic divergence from ID labels using cosine distance in text embedding space.
• Employs a frozen vision-language model for inference, using image and text encoders to compute image-text similarity scores.
• Designs a sum-softmax-based OOD score that normalizes over both ID and negative label affinities, improving discrimination.
• Applies a grouping strategy in inference to stabilize scores and reduce variance without retraining.
• Uses prompt engineering and fixed text encoders to ensure consistent and scalable embedding computation.

Experimental results

Research questions

• RQ1 Can leveraging a large number of negative labels improve OOD detection performance in vision-language models?
• RQ2 How can negative labels with high semantic divergence from ID labels enhance the separability between in-distribution and out-of-distribution samples?
• RQ3 Does the proposed NegLabel method generalize across different vision-language model architectures and benchmarks?
• RQ4 How robust is the method under various domain shifts compared to existing OOD detection approaches?
• RQ5 What is the theoretical basis for the effectiveness of negative labels in OOD detection?

Key findings

• NegLabel achieves 94.21% AUROC and 25.40% FPR95 on the large-scale ImageNet-1k OOD detection benchmark, surpassing many fully-supervised methods.
• The method demonstrates state-of-the-art performance across multiple OOD detection benchmarks, including CIFAR-10, CIFAR-100, and Tiny-ImageNet.
• NegLabel generalizes well across diverse vision-language model architectures, including CLIP, ALIGN, and GroupViT.
• The method exhibits strong robustness against diverse domain shifts, maintaining high performance under distributional shifts.
• Theoretical analysis confirms that the OOD score converges to a normal distribution under large-scale negative label settings, supporting its statistical stability.