[Paper Review] Automatically Discovering and Learning New Visual Categories with Ranking Statistics

Kai Han, Sylvestre-Alvise Rebuffi+3 more|arXiv (Cornell University)|Feb 13, 2020

Domain Adaptation and Few-Shot Learning44 references55 citations

TL;DR

The paper presents a self-supervised, rank-statistics-based approach to automatically discover and learn new visual categories from unlabelled data using labelled data for transfer, with joint training and consistency regularization to avoid forgetting.

ABSTRACT

We tackle the problem of discovering novel classes in an image collection given labelled examples of other classes. This setting is similar to semi-supervised learning, but significantly harder because there are no labelled examples for the new classes. The challenge, then, is to leverage the information contained in the labelled images in order to learn a general-purpose clustering model and use the latter to identify the new classes in the unlabelled data. In this work we address this problem by combining three ideas: (1) we suggest that the common approach of bootstrapping an image representation using the labeled data only introduces an unwanted bias, and that this can be avoided by using self-supervised learning to train the representation from scratch on the union of labelled and unlabelled data; (2) we use rank statistics to transfer the model's knowledge of the labelled classes to the problem of clustering the unlabelled images; and, (3) we train the data representation by optimizing a joint objective function on the labelled and unlabelled subsets of the data, improving both the supervised classification of the labelled data, and the clustering of the unlabelled data. We evaluate our approach on standard classification benchmarks and outperform current methods for novel category discovery by a significant margin.

Motivation & Objective

• Motivate automatic discovery of new visual classes without labeled examples for the new classes.
• Propose a representation learning pipeline that avoids label bias by self-supervision on labeled and unlabeled data.
• Introduce rank statistics to transfer knowledge from labeled to unlabeled data for clustering the new classes.
• Jointly train on labeled and unlabeled data to prevent catastrophic forgetting and improve both supervised and clustering performance.
• Explore incremental learning to extend discovered classes within the classification task.

Proposed method

• Pre-train a CNN representation Phi on the union of labeled and unlabeled data via self-supervised learning (RotNet) to avoid bias from labeled data.
• Fine-tune the last layers on the labeled data to learn a classifier for C^l while keeping the representation robust.
• Use rank statistics to compare unlabelled image representations by top-k activate indices to generate noisy pairwise pseudo-labels s_ij.
• Train a new head eta^u to compute similarity scores for unlabelled pairs with a BCE loss, producing pseudo-label-driven clustering.
• Jointly optimize L_CE (on labeled data) and L_BCE (on unlabelled data) with a consistency regularization term L_MSE to stabilize learning.
• Optionally perform incremental learning by extending the classifier head to include C^u new classes and use pseudo-labels for unlabelled data.

Experimental results

Research questions

• RQ1 Can self-supervision on the combined labeled and unlabeled data produce a more general representation than bootstrapping from labeled data alone?
• RQ2 Can rank-based comparison of unlabeled image embeddings effectively generate pseudo-labels for clustering novel visual categories?
• RQ3 Does joint training on labeled and unlabeled data prevent forgetting and improve both recognition and clustering performance?
• RQ4 Is incremental learning effective for integrating newly discovered categories with existing classes without biasing old-class performance?
• RQ5 How does the proposed method compare to state-of-the-art novel category discovery methods on standard benchmarks?

Key findings

• A self-supervised pretraining on the union of labeled and unlabeled data reduces bias toward known classes and yields robust low-level features.
• Rank statistics using top-k activation sets provide a robust mechanism to generate pseudo-labels for unlabelled data, enabling effective clustering of new categories.
• Jointly optimizing both the supervised loss on labeled data and the clustering loss on unlabeled data with consistency regularization improves performance and mitigates target drift.
• The method achieves state-of-the-art novel category discovery results on CIFAR-10, CIFAR-100, SVHN, OmniGlot, and ImageNet benchmarks.
• Incremental learning further boosts accuracy by enabling a joint classifier over old and new classes, with strong performance on both old and new categories.