[Paper Review] Improving Semantic Segmentation via Self-Training

Yi Zhu, Zhongyue Zhang+7 more|arXiv (Cornell University)|Apr 30, 2020

Advanced Neural Network Applications80 references34 citations

TL;DR

The paper introduces a self-training framework for semantic segmentation that uses a teacher model to generate pseudo labels on unlabeled data and trains a student model with a mix of real and pseudo labels, achieving state-of-the-art results with significantly less supervision and a fast training schedule.

ABSTRACT

Deep learning usually achieves the best results with complete supervision. In the case of semantic segmentation, this means that large amounts of pixelwise annotations are required to learn accurate models. In this paper, we show that we can obtain state-of-the-art results using a semi-supervised approach, specifically a self-training paradigm. We first train a teacher model on labeled data, and then generate pseudo labels on a large set of unlabeled data. Our robust training framework can digest human-annotated and pseudo labels jointly and achieve top performances on Cityscapes, CamVid and KITTI datasets while requiring significantly less supervision. We also demonstrate the effectiveness of self-training on a challenging cross-domain generalization task, outperforming conventional finetuning method by a large margin. Lastly, to alleviate the computational burden caused by the large amount of pseudo labels, we propose a fast training schedule to accelerate the training of segmentation models by up to 2x without performance degradation.

Motivation & Objective

• Reduce reliance on pixel-perfect annotations for semantic segmentation by leveraging unlabeled data.
• Demonstrate that a teacher-student self-training framework can improve performance across multiple driving-scene datasets.
• Address class imbalance and label noise in pseudo labels with centroid sampling.
• Develop a fast training schedule to handle large expanded pseudo-label datasets without performance loss.
• Show cross-domain generalization benefits, including learning with new categories from target domains.

Proposed method

• Train a teacher model on labeled data to produce high-quality pseudo labels for unlabeled images.
• Generate pseudo labels on a large unlabeled set and combine them with real labels to train a student model.
• Use centroid sampling to ensure class-balanced exposure to pseudo labels and mitigate noise.
• Propose fast training schedules that alternate crop sizes to speed up training without sacrificing accuracy (coarse2fine, fine2coarse, and their variants).
• Demonstrate cross-domain generalization by applying self-training to Cityscapes-to-Mapillary with limited target annotations.

Experimental results

Research questions

• RQ1 Can self-training with pseudo labels improve semantic segmentation performance beyond a fully supervised baseline on driving-scene datasets?
• RQ2 Does centroid sampling effectively mitigate label noise and class imbalance in a self-training setting?
• RQ3 Can a fast training schedule maintain accuracy while substantially reducing training time for large pseudo-labeled datasets?
• RQ4 Does self-training improve cross-domain generalization when transferring to a target domain with new categories and limited annotations?

Key findings

• Self-training improves Cityscapes validation mIoU from 78.1 to 79.0 when using Cityscapes coarse and Mapillary pseudo labels.
• Using 1.5K real labels and 1.5K pseudo labels yields 79.3% mIoU, with further gains as pseudo-label quantity increases (e.g., 79.7%, 79.9%, 80.0% in ablations).
• The method generalizes across backbones and architectures, outperforming baselines and matching or exceeding models pre-trained on Mapillary data.
• A fast training schedule (coarse2fine+) delivers up to 1.7–1.8x speedups with no loss in accuracy (vs baseline 80.0% mIoU on Cityscapes validation).
• On cross-domain tasks, self-training with pseudo labels from a source plus limited target annotations achieves competitive results with new categories, outperforming simple finetuning and approaching Mapillary-pretrained performance.