[Paper Review] Mutual Information-guided Knowledge Transfer for Novel Class Discovery

Chuyu Zhang, Chuanyang Hu+4 more|arXiv (Cornell University)|Jun 24, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes a mutual information-guided knowledge transfer method for novel class discovery that explicitly models semantic relationships between seen and unseen classes by maximizing mutual information in a constrained label space. By leveraging mutual information to align semantic knowledge, the approach achieves state-of-the-art performance across multiple benchmarks, significantly outperforming prior methods.

ABSTRACT

We tackle the novel class discovery problem, aiming to discover novel classes in unlabeled data based on labeled data from seen classes. The main challenge is to transfer knowledge contained in the seen classes to unseen ones. Previous methods mostly transfer knowledge through sharing representation space or joint label space. However, they tend to neglect the class relation between seen and unseen categories, and thus the learned representations are less effective for clustering unseen classes. In this paper, we propose a principle and general method to transfer semantic knowledge between seen and unseen classes. Our insight is to utilize mutual information to measure the relation between seen classes and unseen classes in a restricted label space and maximizing mutual information promotes transferring semantic knowledge. To validate the effectiveness and generalization of our method, we conduct extensive experiments both on novel class discovery and general novel class discovery settings. Our results show that the proposed method outperforms previous SOTA by a significant margin on several benchmarks.

Motivation & Objective

• To address the limitation of existing methods in transferring semantic knowledge from seen to unseen classes due to neglect of class relations.
• To improve clustering performance on novel classes by explicitly modeling the semantic relationship between seen and unseen categories.
• To develop a general and principled framework for knowledge transfer that is effective in both standard and general novel class discovery settings.

Proposed method

• The method maximizes mutual information between representations of seen and unseen classes within a restricted label space to capture semantic relationships.
• It formulates knowledge transfer as an optimization problem that encourages alignment of semantic knowledge across seen and unseen classes.
• A contrastive learning objective is employed to enhance feature discrimination while preserving mutual information between seen and unseen class embeddings.
• The framework is designed to be general and applicable to both standard and general novel class discovery settings.
• The method operates in a representation space where mutual information is computed between predicted labels of unseen classes and true labels of seen classes.
• It uses a differentiable approximation of mutual information to enable end-to-end training with gradient-based optimization.

Experimental results

Research questions

• RQ1 How can semantic knowledge be effectively transferred from seen classes to unseen classes in the absence of labeled data for the latter?
• RQ2 To what extent does modeling the class relation between seen and unseen categories improve clustering performance on novel classes?
• RQ3 Can mutual information serve as a reliable and effective inductive bias for knowledge transfer in few-shot and zero-shot class discovery?
• RQ4 How does the proposed method generalize across different benchmarks and settings, including general novel class discovery?
• RQ5 What is the relative contribution of mutual information maximization compared to other knowledge transfer mechanisms like shared representation space or joint label space?

Key findings

• The proposed method achieves state-of-the-art performance on multiple benchmark datasets for novel class discovery.
• It significantly outperforms previous SOTA methods, demonstrating a clear improvement in clustering accuracy on unseen classes.
• The use of mutual information as a knowledge transfer signal leads to more semantically aligned representations between seen and unseen categories.
• The method generalizes well to the general novel class discovery setting, where the number of novel classes is not fixed in advance.
• Empirical results show that mutual information maximization effectively captures inter-class semantic relationships, improving downstream clustering performance.
• Ablation studies confirm that mutual information-based knowledge transfer is more effective than standard representation sharing or joint label space methods.