[Paper Review] PMAL: Open Set Recognition via Robust Prototype Mining

Jing Lü, Yunxu Xu+3 more|arXiv (Cornell University)|Mar 16, 2022

Machine Learning and Data Classification4 citations

TL;DR

This paper proposes PMAL, a novel open set recognition framework that explicitly mines high-quality and diverse prototypes before optimizing the embedding space. By leveraging data uncertainty and diversity-aware filtering, PMAL achieves state-of-the-art performance on multiple benchmarks, significantly improving known-unknown discrimination through robust prototype selection and stable embedding learning.

ABSTRACT

Open Set Recognition (OSR) has been an emerging topic. Besides recognizing predefined classes, the system needs to reject the unknowns. Prototype learning is a potential manner to handle the problem, as its ability to improve intra-class compactness of representations is much needed in discrimination between the known and the unknowns. In this work, we propose a novel Prototype Mining And Learning (PMAL) framework. It has a prototype mining mechanism before the phase of optimizing embedding space, explicitly considering two crucial properties, namely high-quality and diversity of the prototype set. Concretely, a set of high-quality candidates are firstly extracted from training samples based on data uncertainty learning, avoiding the interference from unexpected noise. Considering the multifarious appearance of objects even in a single category, a diversity-based strategy for prototype set filtering is proposed. Accordingly, the embedding space can be better optimized to discriminate therein the predefined classes and between known and unknowns. Extensive experiments verify the two good characteristics (i.e., high-quality and diversity) embraced in prototype mining, and show the remarkable performance of the proposed framework compared to state-of-the-arts.

Motivation & Objective

• Address the limitations of implicit prototype learning in open set recognition, where prototypes are prone to noise and redundancy.
• Improve discrimination between known and unknown classes by explicitly selecting prototypes with high quality and diversity.
• Develop a two-stage framework—prototype mining followed by embedding learning—that enhances model robustness and generalization.
• Stabilize prototype selection across different pre-trained models to ensure reliable and consistent mining.
• Demonstrate that explicit prototype mining leads to better intra-class compactness and inter-class separation in the embedding space.

Proposed method

• Introduce a novel metric, embedding topology robustness, to identify high-quality training samples by measuring data uncertainty from low-quality factors such as occlusion or blur.
• Apply a diversity-based filtering strategy to select prototypes per class, reducing redundancy and preserving multifarious appearances within each category.
• Use the selected high-quality and diverse prototypes as fixed anchors in a point-to-set distance metric during embedding optimization.
• Optimize the embedding space using a joint loss combining classification loss and a prototype-aware margin loss to enforce separation between known and unknown classes.
• Employ multiple pre-trained models (U=2) to compute prototype quality scores, ensuring stability and reducing dependency on a single model.
• Maintain inference efficiency by using only one model for feature extraction during inference, avoiding additional parameters.

Experimental results

Research questions

• RQ1 Can explicit prototype mining with quality and diversity constraints improve open set recognition performance compared to implicit prototype learning?
• RQ2 How does the robustness of prototype selection vary across different pre-trained models, and does it ensure stable mining results?
• RQ3 To what extent does diversity in prototype selection enhance intra-class compactness and inter-class separation in the embedding space?
• RQ4 How do hyperparameters like ε (quality threshold), δ (margin), and λp (loss weight) affect the model’s AUROC and convergence?
• RQ5 Does the proposed framework outperform state-of-the-art methods on diverse benchmarks, especially in complex datasets like TinyImageNet?

Key findings

• PMAL achieves state-of-the-art AUROC performance on both MNIST and TinyImageNet benchmarks, with a 10.2% improvement over the previous SOTA on TinyImageNet.
• The diversity-based prototype selection method outperforms random selection and K-Means clustering, as shown in Table 5, with consistent gains across multiple datasets.
• Prototype mining is stable across different pre-trained models, with minimal fluctuation in prototype positions across three repeated runs, indicating robustness.
• Using T=10 prototypes per class leads to more compact intra-class embeddings than T=1, reducing overlap with unknown samples and improving generalization.
• The method is relatively insensitive to hyperparameter λp, with AUROC varying by only 0.3 points across λp values of 0.5 to 1.0, indicating good stability.
• The embedding space learned by PMAL shows significantly larger separation between known classes and unknowns compared to CPN and RPL, especially on complex datasets like TinyImageNet.