[Paper Review] Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels

Yuchao Wang, Haochen Wang+7 more|arXiv (Cornell University)|Mar 8, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes U2PL, a semi-supervised semantic segmentation framework that leverages unreliable pseudo-labels by treating ambiguous predictions as negative samples for unlikely classes. By separating reliable and unreliable pixels via entropy, maintaining category-wise memory banks for negative samples, and adaptively adjusting the reliability threshold, U2PL achieves state-of-the-art performance on PASCAL VOC 2012 and Cityscapes, outperforming prior methods by up to 2.2% mIoU under low-shot settings.

ABSTRACT

The crux of semi-supervised semantic segmentation is to assign adequate pseudo-labels to the pixels of unlabeled images. A common practice is to select the highly confident predictions as the pseudo ground-truth, but it leads to a problem that most pixels may be left unused due to their unreliability. We argue that every pixel matters to the model training, even its prediction is ambiguous. Intuitively, an unreliable prediction may get confused among the top classes (i.e., those with the highest probabilities), however, it should be confident about the pixel not belonging to the remaining classes. Hence, such a pixel can be convincingly treated as a negative sample to those most unlikely categories. Based on this insight, we develop an effective pipeline to make sufficient use of unlabeled data. Concretely, we separate reliable and unreliable pixels via the entropy of predictions, push each unreliable pixel to a category-wise queue that consists of negative samples, and manage to train the model with all candidate pixels. Considering the training evolution, where the prediction becomes more and more accurate, we adaptively adjust the threshold for the reliable-unreliable partition. Experimental results on various benchmarks and training settings demonstrate the superiority of our approach over the state-of-the-art alternatives.

Motivation & Objective

• To address the underutilization of unlabeled data in semi-supervised semantic segmentation, especially pixels with low-confidence predictions.
• To overcome the limitation of existing methods that discard unreliable predictions, which may lead to class-imbalanced and insufficient training.
• To propose a framework that effectively incorporates unreliable predictions not as positive pseudo-labels, but as negative signals for unlikely classes.
• To improve model generalization on ambiguous regions such as object boundaries by leveraging all pixels, including those with high entropy.

Proposed method

• The method uses pixel-wise entropy to classify predictions as reliable (low entropy) or unreliable (high entropy), enabling separation of high-confidence and ambiguous predictions.
• Reliable predictions are used as positive pseudo-labels for supervised training, while unreliable predictions are treated as negative samples for their least probable classes.
• A category-wise memory bank is maintained for each class, storing unreliable predictions as negative samples to balance class representation and improve contrastive learning.
• An adaptive thresholding strategy dynamically adjusts the reliable-unreliable partition during training based on model confidence evolution, improving robustness and generalization.
• The framework integrates a contrastive loss with a category-wise memory bank to enhance feature discrimination, using both reliable and unreliable pixels.
• A probability rank threshold (PRT) and high-entropy filtering are applied to refine the selection of negative samples, ensuring only the most informative unreliable predictions are used.

Figure 1 : Category-wise performance and statistics on number of pixels with reliable and unreliable predictions. Model is trained using $732$ labeled images on PASCAL VOC 2012 [ 14 ] and evaluated on the remaining $9,850$ images.

Experimental results

Research questions

• RQ1 Can unreliable pseudo-labels, traditionally discarded, be effectively leveraged in semi-supervised semantic segmentation?
• RQ2 How can ambiguous predictions be transformed into useful training signals without introducing noise or bias?
• RQ3 Does maintaining a category-wise memory bank for negative samples improve model generalization and reduce class imbalance?
• RQ4 Can adaptive thresholding based on prediction entropy improve training efficiency and performance over fixed thresholds?
• RQ5 Does incorporating unreliable predictions lead to better segmentation performance on ambiguous regions like object boundaries?

Key findings

• U2PL achieves a mean Intersection over Union (mIoU) of 79.30% on the PASCAL VOC 2012 validation set under the 1/4 partition protocol, surpassing previous state-of-the-art methods.
• The ablation study shows that combining all components—including contrastive loss, category-wise memory banks, dynamic partition adjustment, and high-entropy filtering—leads to a 6.05% improvement over the baseline.
• Without high-entropy filtering, the performance gain drops significantly to only +4.55%, indicating the critical role of filtering unreliable predictions before negative sampling.
• The optimal initial reliable-unreliable partition ratio is found to be 20% reliable predictions, as other ratios (e.g., 10% or 40%) degrade performance.
• Qualitative results demonstrate that U2PL significantly improves segmentation accuracy on ambiguous regions such as object borders, where traditional methods struggle.

Figure 2 : Illustration on unreliable pseudo-labels. (a) Pixel-wise entropy predicted from an unlabeled image, where low-entropy pixels and high-entropy pixels indicate reliable and unreliable predictions, respectively. (b) Pixel-wise pseudo-labels from reliable predictions only , where pixels withi