[Paper Review] Semi-supervised semantic segmentation needs strong, varied perturbations

Geoff French, Samuli Laine+3 more|arXiv (Cornell University)|Jun 5, 2019

Domain Adaptation and Few-Shot Learning39 references162 citations

TL;DR

The paper analyzes why consistency regularization struggles in semantic segmentation and shows that strong, varied, mask-based augmentations (adapted CutOut/CutMix) yield state-of-the-art semi-supervised segmentation on natural images within a mean-teacher framework.

ABSTRACT

Consistency regularization describes a class of approaches that have yielded ground breaking results in semi-supervised classification problems. Prior work has established the cluster assumption - under which the data distribution consists of uniform class clusters of samples separated by low density regions - as important to its success. We analyze the problem of semantic segmentation and find that its' distribution does not exhibit low density regions separating classes and offer this as an explanation for why semi-supervised segmentation is a challenging problem, with only a few reports of success. We then identify choice of augmentation as key to obtaining reliable performance without such low-density regions. We find that adapted variants of the recently proposed CutOut and CutMix augmentation techniques yield state-of-the-art semi-supervised semantic segmentation results in standard datasets. Furthermore, given its challenging nature we propose that semantic segmentation acts as an effective acid test for evaluating semi-supervised regularizers. Implementation at: https://github.com/Britefury/cutmix-semisup-seg.

Motivation & Objective

• Assess the data distribution of semantic segmentation and its impact on consistency regularization.
• Identify whether low-density regions separate classes in pixel-level segmentation.
• Develop and evaluate strong perturbation augmentations suitable for segmentation.
• Demonstrate state-of-the-art semi-supervised segmentation results using adapted CutMix in a mean-teacher setup.

Proposed method

• Analyze patch-level data density for segmentation to assess cluster assumption validity.
• Adapt CutOut and CutMix augmentations for semi-supervised segmentation with a mean teacher model.
• Use a pixel-wise consistency loss that compares predictions on perturbed inputs to teacher-generated targets.
• Employ a variance-rich augmentation strategy to constrain decision boundaries without relying on low-density regions.
• Evaluate on Cityscapes, augmented Pascal VOC, and ISIC 2017 to compare perturbation effectiveness.

Experimental results

Research questions

• RQ1 Does semantic segmentation exhibit low-density regions separating classes as in standard classification, affecting consistency regularization?
• RQ2 Can strong, varied, mask-based augmentations enable effective semi-supervised learning for segmentation in the absence of clear low-density regions?
• RQ3 How do adapted CutOut and CutMix augmentations perform in semi-supervised segmentation compared to other perturbations across natural and medical image datasets?

Key findings

• Semantic segmentation lacks clear low-density inter-class regions, challenging traditional consistency regularization.
• Adapted CutOut and CutMix provide strong, varied perturbations that yield state-of-the-art semi-supervised segmentation results on natural image datasets.
• Mean teacher framework with these augmentations outperforms several adversarial and standard augmentation baselines on Cityscapes and augmented Pascal VOC.
• CutMix typically delivers the strongest gains among the proposed perturbations across tested datasets.
• ISIC 2017 results show CutMix as the most reliable augmentation for semi-supervised lesion segmentation among the tested methods.