[Paper Review] Open-Set Recognition: a Good Closed-Set Classifier is All You Need?

Sagar Vaze, Kai Han+2 more|arXiv (Cornell University)|Oct 12, 2021

Domain Adaptation and Few-Shot LearningComputer Science61 references40 citations

TL;DR

The paper shows a strong empirical link between closed-set accuracy and open-set recognition (OSR) performance, and demonstrates that improving closed-set accuracy with standard image-classification techniques yields state-of-the-art OSR results, including on large-scale ImageNet splits. It also introduces the Semantic Shift Benchmark (SSB) to better evaluate semantic novelty in OSR.

ABSTRACT

The ability to identify whether or not a test sample belongs to one of the semantic classes in a classifier's training set is critical to practical deployment of the model. This task is termed open-set recognition (OSR) and has received significant attention in recent years. In this paper, we first demonstrate that the ability of a classifier to make the 'none-of-above' decision is highly correlated with its accuracy on the closed-set classes. We find that this relationship holds across loss objectives and architectures, and further demonstrate the trend both on the standard OSR benchmarks as well as on a large-scale ImageNet evaluation. Second, we use this correlation to boost the performance of a maximum logit score OSR 'baseline' by improving its closed-set accuracy, and with this strong baseline achieve state-of-the-art on a number of OSR benchmarks. Similarly, we boost the performance of the existing state-of-the-art method by improving its closed-set accuracy, but the resulting discrepancy with the strong baseline is marginal. Our third contribution is to present the 'Semantic Shift Benchmark' (SSB), which better respects the task of detecting semantic novelty, in contrast to other forms of distribution shift also considered in related sub-fields, such as out-of-distribution detection. On this new evaluation, we again demonstrate that there is negligible difference between the strong baseline and the existing state-of-the-art. Project Page: https://www.robots.ox.ac.uk/~vgg/research/osr/

Motivation & Objective

• Demonstrate that closed-set performance strongly correlates with open-set detection across datasets and architectures.
• Show that improving closed-set accuracy of MSP baseline yields state-of-the-art OSR results.
• Propose a unified, semantics-aware evaluation suite (Semantic Shift Benchmark) for OSR beyond traditional openness measures.

Proposed method

• Compare MSP baseline, ARPL, and ARPL+CS on standard OSR benchmarks across multiple datasets.
• Quantify correlation between closed-set accuracy and open-set AUROC across datasets and architectures.
• Improve the MSP baseline using longer training, augmented data, and label smoothing to raise closed-set accuracy (MSP+).
• Propose maximum logit score (MLS) as an open-set indicator instead of softmax probabilities.
• Evaluate MLS and strengthened baselines on large-scale ImageNet splits with easy/hard semantic open-set sets.
• Introduce and evaluate the Semantic Shift Benchmark (SSB) comprising ImageNet-scale and fine-grained FGVC datasets to assess semantic novelty.

Experimental results

Research questions

• RQ1 Does closed-set accuracy correlate with open-set detection performance across datasets and model families?
• RQ2 Can improving closed-set accuracy of a baseline OSR method yield competitive or superior OSR performance compared to state-of-the-art methods?
• RQ3 How does an open-set scoring rule based on maximum logits (MLS) compare to maximum softmax probability (MSP) for OSR?
• RQ4 What is the impact of semantic-aware open-set splits on OSR evaluation compared to openness alone?
• RQ5 Does the proposed Semantic Shift Benchmark provide a meaningful, semantics-centered OSR evaluation framework at large scale?

Key findings

• There is a strong positive correlation between closed-set accuracy and open-set AUROC across benchmarks (Pearson ρ ≈ 0.95 on standard benchmarks, and ≈ 0.88/0.63 on ImageNet Easy/Hard splits).
• Strengthening the MSP baseline with standard image-classification improvements yields state-of-the-art or competitive OSR results on most benchmarks (e.g., MSP+ and MLS outperform several baselines).
• Maximum Logit Score (MLS) as an open-set indicator provides significant gains over MSP baseline, with MLS achieving superior average AUROC across datasets.
• On the Semantic Shift Benchmark, MLS and ARPL+ show comparable performance, underscoring that semantic-awareness splits matter for OSR evaluation.
• The proposed Semantic Shift Benchmark reveals that semantically harder splits reduce OSR performance more than simple openness measures would suggest.