[Paper Review] Unseen Class Discovery in Open-world Classification

Lei Shu, Hu Xu+1 more|arXiv (Cornell University)|Jan 17, 2018

Machine Learning and Data Classification24 references57 citations

TL;DR

The paper presents a joint Open Classification Network (OCN) and Pairwise Classification Network (PCN) that, using seen-class knowledge, discovers and clusters unseen classes among rejected open-world samples via hierarchical clustering guided by a learned same/different pair distance.

ABSTRACT

This paper concerns open-world classification, where the classifier not only needs to classify test examples into seen classes that have appeared in training but also reject examples from unseen or novel classes that have not appeared in training. Specifically, this paper focuses on discovering the hidden unseen classes of the rejected examples. Clearly, without prior knowledge this is difficult. However, we do have the data from the seen training classes, which can tell us what kind of similarity/difference is expected for examples from the same class or from different classes. It is reasonable to assume that this knowledge can be transferred to the rejected examples and used to discover the hidden unseen classes in them. This paper aims to solve this problem. It first proposes a joint open classification model with a sub-model for classifying whether a pair of examples belongs to the same or different classes. This sub-model can serve as a distance function for clustering to discover the hidden classes of the rejected examples. Experimental results show that the proposed model is highly promising.

Motivation & Objective

• Address open-world classification where test data can belong to unseen classes not seen during training.
• Enable discovery of hidden unseen classes among rejected samples from an open classifier without manual labeling.
• Transfer similarity knowledge learned from seen classes to unseen classes to guide clustering.

Proposed method

• Train a joint model consisting of an Open Classification Network (OCN) and a Pairwise Classification Network (PCN) with a shared representation.
• OCN performs standard open classification with rejection for unseen classes using a 1-vs-rest sigmoid output (DOC-based).
• PCN learns a binary same/different classifier for pairs of examples using two identical CNN branches.
• A hierarchical clustering (complete-linkage) uses the PCN distance as the clustering metric to discover unseen class clusters.
• An auto-encoder is trained alongside to provide unsupervised representations and prevent overfitting to seen classes.
• A validation-based threshold theta is used to determine the stopping criterion for clustering, ensuring the correct number of unseen clusters.
• Training data: (i) open-classification data for OCN, (ii) seen-seen pairs for PCN, (iii) unlabeled data for auto-encoder; losses L_OCN, L_PCN, and L_ae are jointly optimized.

Experimental results

Research questions

• RQ1 Can an open-world classifier reject unseen class samples and provide a meaningful distance function for clustering unseen samples?
• RQ2 Can a transferred similarity (distance) function learned from seen classes guide hierarchical clustering to reveal the number of hidden unseen classes?
• RQ3 How well does the combined OCN+PCN+HC pipeline discover and cluster unseen classes in open-world settings?

Key findings

• OCN consistently outperforms OpenMax in rejecting unseen class samples on MNIST and EMNIST datasets.
• PCN transfers learned intra-/inter-class similarity from seen classes to unseen-class pairs, achieving reasonable accuracy on seen-seen, seen-unseen, and unseen-unseen pairs.
• Hierarchical clustering using PCN as the distance function can almost recover the true number of unseen clusters on MNIST and EMNIST unseen sets; the approach remains robust when clustering rejected (noisy) samples.
• Compared to Encoder+HC, PCN+HC obtains substantially better cluster-count accuracy, especially on EMNIST, demonstrating the benefit of the distance learned by PCN.
• K-means can achieve higher clustering quality given the true number of clusters but cannot infer the number of clusters, whereas PCN+HC can estimate the cluster count and then enable K-means to refine clusters.