[Paper Review] Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision

Xiaokang Chen, Yuhui Yuan+2 more|arXiv (Cornell University)|Jun 2, 2021

Advanced Neural Network ApplicationsComputer Science38 references59 citations

TL;DR

Introduces cross pseudo supervision (CPS) to train two independently initialized segmentation networks with mutual pseudo-labels on unlabeled data, achieving state-of-the-art semi-supervised segmentation on Cityscapes and PASCAL VOC 2012.

ABSTRACT

In this paper, we study the semi-supervised semantic segmentation problem via exploring both labeled data and extra unlabeled data. We propose a novel consistency regularization approach, called cross pseudo supervision (CPS). Our approach imposes the consistency on two segmentation networks perturbed with different initialization for the same input image. The pseudo one-hot label map, output from one perturbed segmentation network, is used to supervise the other segmentation network with the standard cross-entropy loss, and vice versa. The CPS consistency has two roles: encourage high similarity between the predictions of two perturbed networks for the same input image, and expand training data by using the unlabeled data with pseudo labels. Experiment results show that our approach achieves the state-of-the-art semi-supervised segmentation performance on Cityscapes and PASCAL VOC 2012. Code is available at https://git.io/CPS.

Motivation & Objective

• Motivate semi-supervised semantic segmentation using both labeled and unlabeled data.
• Propose a simple consistency regularization via cross pseudo supervision between two independently initialized networks.
• Show that cross supervision expands effective training data with pseudo labels while enforcing cross-network consistency.
• Demonstrate state-of-the-art performance on Cityscapes and PASCAL VOC 2012 under various labeled/unlabeled partitions.

Proposed method

• Two segmentation networks with same structure but different initializations process the same input.
• Each network outputs a probability/confidence map; one-hot pseudo segmentation maps are derived from these outputs.
• Cross pseudo supervision losses supervise the other network using the peer network's pseudo labels in a bidirectional manner.
• Loss comprises a standard supervision term on labeled data and a cross pseudo supervision term on both labeled and unlabeled data (L = Ls + lambda * Lcps).
• Incorporates CutMix augmentation to generate mixed inputs and corresponding mixed pseudo labels for training.
• Discusses variants such as cross probability consistency, mean-teacher, and single-network pseudo supervision for comparison.

Figure 1: Illustrating the architectures for (a) our approach cross pseudo supervision, (b) cross confidence consistency (e.g., a component of GCT [ 17 ] ), (c) mean teacher (used in CutMix-Seg [ 11 ] ), and (d) PseudoSeg [ 44 ] structure (similar to FixMatch [ 28 ] ). ‘ $\rightarrow$ ’ means forwar

Experimental results

Research questions

• RQ1 Can two independently initialized segmentation networks provide mutually supervising pseudo labels to improve semi-supervised semantic segmentation?
• RQ2 Does cross pseudo supervision outperform existing consistency-based or self-training approaches on standard benchmarks?
• RQ3 What is the impact of applying cross pseudo supervision to labeled versus unlabeled data, and what is the effect of CutMix augmentation?
• RQ4 How does CPS interact with self-training, and can their combination further improve performance?

Key findings

• CPS consistently improves over supervised baselines across Cityscapes and PASCAL VOC 2012 under multiple labeling ratios.
• Applying cross pseudo supervision to unlabeled data yields larger gains than applying it only to labeled data.
• CPS outperforms cross probability consistency and mean-teacher baselines in the reported experiments.
• CutMix augmentation enhances CPS, with larger gains at lower amounts of labeled data.
• Combining CPS with self-training yields further improvements over either method alone.
• Even with full supervision, CPS can still benefit from unlabeled data in training.

Figure 6: Example qualitative results from PASCAL VOC $2012$ . (a) input, (b) ground truth, (c) supervised only, (d) ours (w/o CutMix Aug.), and (e) ours (w/ CutMix Aug.). All the approaches use DeepLabv $3$ + with ResNet- $101$ as the segmentation network.