[Paper Review] Re-distributing Biased Pseudo Labels for Semi-supervised Semantic Segmentation: A Baseline Investigation

Ruifei He, Jihan Yang+1 more|arXiv (Cornell University)|Jul 23, 2021

Advanced Neural Network Applications63 references5 citations

TL;DR

This paper proposes DARS, a simple yet effective method to re-distribute biased pseudo-labels in semi-supervised semantic segmentation by aligning their class distribution with the true distribution from labeled data. Using class-wise thresholding and random sampling, DARS corrects the long-tailed bias in self-training, achieving 8.89% mIoU gain on Cityscapes (1/8 split) and 4.49% on PASCAL VOC 2012, outperforming state-of-the-art methods.

ABSTRACT

While self-training has advanced semi-supervised semantic segmentation, it severely suffers from the long-tailed class distribution on real-world semantic segmentation datasets that make the pseudo-labeled data bias toward majority classes. In this paper, we present a simple and yet effective Distribution Alignment and Random Sampling (DARS) method to produce unbiased pseudo labels that match the true class distribution estimated from the labeled data. Besides, we also contribute a progressive data augmentation and labeling strategy to facilitate model training with pseudo-labeled data. Experiments on both Cityscapes and PASCAL VOC 2012 datasets demonstrate the effectiveness of our approach. Albeit simple, our method performs favorably in comparison with state-of-the-art approaches. Code will be available at https://github.com/CVMI-Lab/DARS.

Motivation & Objective

• Address the severe class distribution mismatch in self-training-based semi-supervised semantic segmentation, where pseudo-labels are biased toward majority classes due to long-tailed data and model overconfidence.
• Identify that single-threshold pseudo-labeling fails under long-tailed distributions, especially for tail classes, due to confidence overlapping and biased predictions.
• Propose a distribution alignment method that matches pseudo-labels to the true class distribution from labeled data, overcoming bias without relying on confidence thresholds alone.
• Introduce a progressive data augmentation and labeling strategy to stabilize training and prevent overfitting to easy, high-confidence examples in early stages.
• Demonstrate that the method is generic, efficient, and effective, achieving state-of-the-art performance with minimal code changes.

Proposed method

• Propose Distribution Alignment and Random Sampling (DARS), which uses class-wise thresholds and random sampling to re-distribute pseudo-labels so their class distribution matches the true distribution from labeled data.
• Address confidence overlapping in deep networks by avoiding reliance on a single global threshold, instead applying per-class confidence thresholds to achieve precise distribution alignment.
• Introduce a progressive data augmentation and labeling strategy that gradually increases augmentation strength and labeling ratio over training epochs to improve robustness and generalization.
• Use PSPNet and DeepLabv2 as backbone models, applying DARS to generate unbiased pseudo-labels during iterative self-training.
• Estimate the true class distribution from the labeled set and use it as a target for aligning pseudo-label distributions, minimizing distribution mismatch.
• Apply the method in both low-data (1/8 split) and high-data regimes, including fully supervised fine-tuning with extra pseudo-labeled data.

Experimental results

Research questions

• RQ1 Can re-distributing biased pseudo-labels to match the true class distribution significantly improve semi-supervised semantic segmentation performance?
• RQ2 Does the use of class-wise thresholding and random sampling effectively correct distribution mismatch caused by long-tailed data and model bias?
• RQ3 How does progressive data augmentation and labeling affect model convergence and robustness during self-training?
• RQ4 What are the main bottlenecks limiting performance gains in high-data regimes for semi-supervised semantic segmentation?
• RQ5 Can a simple, generic method like DARS outperform more complex, state-of-the-art approaches in semi-supervised semantic segmentation?

Key findings

• DARS achieves a +8.89% mIoU improvement on Cityscapes under the 1/8 split setting, significantly outperforming previous methods.
• On PASCAL VOC 2012 with 1.4k labeled and 9k augmented unlabeled images, DARS improves mIoU by 4.49% over the previous state of the art.
• Performance gains from DARS are most significant for tail classes, especially those with rare occurrence, while head classes show minimal improvement.
• In high-data regimes, performance gains saturate due to two main bottlenecks: (1) originally rare classes evolve into head classes, and (2) systematic errors in deep networks toward small-sized objects limit further gains.
• The network's accuracy decreases significantly for small-sized objects, indicating that systematic error for small objects is a key bottleneck in high-data semi-supervised learning.
• Extending fully supervised models with 9K extra pseudo-labeled images from DARS yields only a 2.1% mIoU gain, suggesting performance saturation in high-data regimes.