[Paper Review] Virtual Category Learning: A Semi-Supervised Learning Method for Dense Prediction with Extremely Limited Labels

Changrui Chen, Jungong Han+1 more|arXiv (Cornell University)|Dec 2, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes Virtual Category (VC) learning, a semi-supervised method for dense prediction tasks that assigns virtual categories to confusing, hard-to-label unlabelled samples instead of discarding or directly using noisy pseudo-labels. By enabling safe optimization via these virtual categories, the method achieves state-of-the-art performance—especially under extremely limited supervision—surpassing existing approaches on semantic segmentation and object detection with minimal labelled data.

ABSTRACT

Due to the costliness of labelled data in real-world applications, semi-supervised learning, underpinned by pseudo labelling, is an appealing solution. However, handling confusing samples is nontrivial: discarding valuable confusing samples would compromise the model generalisation while using them for training would exacerbate the issue of confirmation bias caused by the resulting inevitable mislabelling. To solve this problem, this paper proposes to use confusing samples proactively without label correction. Specifically, a Virtual Category (VC) is assigned to each confusing sample in such a way that it can safely contribute to the model optimisation even without a concrete label. This provides an upper bound for inter-class information sharing capacity, which eventually leads to a better embedding space. Extensive experiments on two mainstream dense prediction tasks -- semantic segmentation and object detection, demonstrate that the proposed VC learning significantly surpasses the state-of-the-art, especially when only very few labels are available. Our intriguing findings highlight the usage of VC learning in dense vision tasks.

Motivation & Objective

• To address the challenge of training deep models with extremely limited labelled data in dense prediction tasks such as semantic segmentation and object detection.
• To overcome the limitations of existing pseudo-labeling methods that either discard confusing samples or suffer from confirmation bias due to noisy labels.
• To enable effective utilization of hard, ambiguous unlabelled samples without requiring ground-truth labels by introducing a virtual category mechanism.
• To improve model generalization and optimization stability in low-shot semi-supervised learning scenarios.

Proposed method

• Assign a Virtual Category (VC) to each confusing unlabelled sample based on its pseudo-label distribution, rather than relying on a single potentially incorrect label.
• Construct a Potential Category (PC) set for each confusing sample, representing all plausible classes it might belong to.
• Introduce a modified loss function, Reg* loss, that encourages model optimization along a safe, inter-class boundary direction even without ground-truth labels.
• Decouple the VC loss from the main training graph via gradient detachment when ablation is needed, enabling ablation studies on VC contribution.
• Apply VC learning selectively in later stages of detection networks (e.g., RPN and region classifier) to preserve performance while improving robustness.
• Use a confidence threshold and temporal stability verification to filter out unreliable pseudo-labels before assigning VCs.

Experimental results

Research questions

• RQ1 Can confusing unlabelled samples be effectively utilized in semi-supervised dense prediction without ground-truth labels?
• RQ2 Does assigning a virtual category to ambiguous samples improve model generalization compared to discarding or directly using noisy pseudo-labels?
• RQ3 How does VC learning perform under extremely limited supervision, especially when baseline models are weak?
• RQ4 Can VC learning be effectively applied in both semantic segmentation and object detection tasks?
• RQ5 What is the contribution of VC learning versus simple filtering or loss omission in model optimization?

Key findings

• VC learning achieves state-of-the-art performance on semantic segmentation with only 1/128 of Pascal VOC labels, outperforming existing methods.
• On 1% labelled MS COCO, VC learning achieves 20.81 mAP, significantly outperforming baseline methods and strategies that discard or retain all confusing samples.
• Ablation studies confirm that VC learning delivers the best performance, with 'discard' and 'detach' variants yielding worse results, proving VC's contribution is not merely due to loss component omission.
• Applying VC learning in both RPN and region classifier stages improves mAP from 20.00 to 20.81, demonstrating its effectiveness across detection stages.
• On miniImageNet, VC learning achieves 51.49% top-1 accuracy, surpassing existing methods by a large margin in a non-dense classification setting.
• The method remains effective even with very few labels, showing diminishing returns as label ratio increases, which is expected and consistent with theoretical expectations.