[论文解读] Class-Conditional Conformal Prediction with Many Classes

Tiffany Ding, Anastasios N. Angelopoulos+3 more|arXiv (Cornell University)|Jun 15, 2023

Domain Adaptation and Few-Shot Learning被引用 11

一句话总结

该论文开发了聚类条件预测，以在多类、数据有限的设置中实现类别条件覆盖，通过将具有相似分数分布的类别进行聚类，并在聚类层面应用条件校准。

ABSTRACT

Standard conformal prediction methods provide a marginal coverage guarantee, which means that for a random test point, the conformal prediction set contains the true label with a user-specified probability. In many classification problems, we would like to obtain a stronger guarantee--that for test points of a specific class, the prediction set contains the true label with the same user-chosen probability. For the latter goal, existing conformal prediction methods do not work well when there is a limited amount of labeled data per class, as is often the case in real applications where the number of classes is large. We propose a method called clustered conformal prediction that clusters together classes having "similar" conformal scores and performs conformal prediction at the cluster level. Based on empirical evaluation across four image data sets with many (up to 1000) classes, we find that clustered conformal typically outperforms existing methods in terms of class-conditional coverage and set size metrics.

研究动机与目标

• Motivate the need for class-conditional coverage in multi-class classification.
• Propose clustered conformal prediction to handle limited per-class data.
• Demonstrate empirical improvements over standard and classwise conformal prediction on large-scale image datasets.
• Provide practical guidelines for selecting conformal methods in many-classes settings.

提出的方法

• Extend conformal prediction to cluster classes with similar score distributions.
• Split calibration data into clustering and proper calibration sets.
• Compute cluster-level conformal quantiles and assign prediction sets based on the cluster of each class.
• Include a null cluster for rare classes and fallback to standard calibration for them.
• Embed class score distributions into a quantile-based feature space to drive clustering via k-means.
• Provide theoretical guarantees: cluster-conditional coverage for non-null clusters and approximate class-conditional coverage under an oracle clustering.

实验结果

研究问题

• RQ1 Can clustering classes by score distribution improve class-conditional coverage under limited calibration data?
• RQ2 Does clustered conformal prediction balance coverage and set size better than standard and classwise methods in many-class settings?
• RQ3 How should practitioners choose clustering fraction and number of clusters for practical performance?
• RQ4 What are the theoretical guarantees when using clustered conformal prediction with an approximate (non-oracle) clustering?
• RQ5 How does clustered conformal prediction perform across diverse large-scale datasets with up to 1000 classes?

主要发现

• Clustered conformal prediction often achieves the best or comparable class-coverage gaps across datasets and score functions.
• Clustered generally outperforms standard and classwise in limited-data regimes, especially at moderate calibration sizes, and achieves competitive set sizes.
• Classwise can be unstable with few calibration examples per class, while standard lacks class-conditional guarantees; clustered mitigates both weaknesses.
• Empirical results show improvements on ImageNet, CIFAR-100, Places365, and iNaturalist across softmax, APS, and RAPS scores.
• The method provides practical guidelines for choosing gamma and M to balance cluster-conditional coverage and data efficiency.