[Paper Review] Parametric Classification for Generalized Category Discovery: A Baseline Study

Xin Wen, Bingchen Zhao+1 more|arXiv (Cornell University)|Nov 21, 2022

Machine Learning and Data Classification4 citations

TL;DR

This paper proposes a simple yet effective parametric classification method for Generalized Category Discovery (GCD) that overcomes the failure of prior parametric classifiers by addressing unreliable pseudo-labels and prediction biases. By incorporating entropy regularization and self-distillation, the method achieves state-of-the-art performance on multiple GCD benchmarks while maintaining robustness to unknown numbers of novel categories.

ABSTRACT

Generalized Category Discovery (GCD) aims to discover novel categories in unlabelled datasets using knowledge learned from labelled samples. Previous studies argued that parametric classifiers are prone to overfitting to seen categories, and endorsed using a non-parametric classifier formed with semi-supervised k-means. However, in this study, we investigate the failure of parametric classifiers, verify the effectiveness of previous design choices when high-quality supervision is available, and identify unreliable pseudo-labels as a key problem. We demonstrate that two prediction biases exist: the classifier tends to predict seen classes more often, and produces an imbalanced distribution across seen and novel categories. Based on these findings, we propose a simple yet effective parametric classification method that benefits from entropy regularisation, achieves state-of-the-art performance on multiple GCD benchmarks and shows strong robustness to unknown class numbers. We hope the investigation and proposed simple framework can serve as a strong baseline to facilitate future studies in this field. Our code is available at: https://github.com/CVMI-Lab/SimGCD.

Motivation & Objective

• To investigate why previous parametric classifiers fail in Generalized Category Discovery (GCD), particularly in recognizing novel categories.
• To identify unreliable pseudo-labels and prediction biases—such as over-prediction of seen classes and imbalanced label distribution—as the root cause of poor performance.
• To develop a simple, effective parametric classification baseline that outperforms non-parametric alternatives while maintaining low inference cost.
• To demonstrate that entropy regularization can improve prediction balance and robustness, especially under unknown class numbers.
• To establish a strong, reproducible baseline for future GCD research, challenging the recent trend toward non-parametric methods.

Proposed method

• Adopts a parametric classifier jointly trained with a backbone network using cross-entropy loss on labelled data.
• Applies self-distillation to unlabelled data to generate soft pseudo-labels, improving generalization without explicit clustering.
• Introduces an entropy regularization term to encourage uniform prediction distribution across all categories, counteracting bias toward seen classes.
• Uses the same representation learning objective as prior work (GCD [43]), focusing on improving classification rather than representation learning.
• Employs a joint optimization scheme that alternates between supervised learning on labelled data and self-distillation on unlabelled data.
• Treats both seen and novel categories as part of a unified classification head, enabling end-to-end learning with balanced prediction targets.

Experimental results

Research questions

• RQ1 Why do parametric classifiers underperform in GCD despite strong supervision and representation learning?
• RQ2 What specific biases in model predictions lead to poor generalization to novel categories in GCD?
• RQ3 Can entropy regularization effectively mitigate prediction imbalance and improve performance on unseen classes?
• RQ4 How does the proposed method compare to non-parametric classifiers like k-means in terms of accuracy, inference cost, and robustness?
• RQ5 To what extent can entropy regularization enhance robustness when the number of novel categories is unknown?

Key findings

• Previous parametric classifiers fail primarily due to unreliable pseudo-labels and strong prediction biases favoring seen categories.
• The proposed method achieves state-of-the-art performance on multiple GCD benchmarks, including CIFAR-100, CUB, and Tiny-ImageNet, outperforming prior parametric and non-parametric approaches.
• Entropy regularization successfully produces more balanced pseudo-label distributions, as evidenced by visualizations in Figures 9 and 10.
• The model demonstrates strong robustness to unknown numbers of novel categories, a key advantage in real-world deployment.
• A trade-off between performance on seen and novel categories is observed during training, highlighting the need for balanced optimization.
• The method maintains low inference cost compared to non-parametric k-means clustering, which suffers from quadratic complexity on large-scale datasets.