[Paper Review] Semi-supervised semantic segmentation needs strong, high-dimensional perturbations

Geoff French, Timo Aila+3 more|arXiv (Cornell University)|Sep 25, 2019

Domain Adaptation and Few-Shot Learning17 references74 citations

TL;DR

This paper identifies the lack of low-density regions between semantic classes as a key reason semi-supervised semantic segmentation underperforms compared to classification. It demonstrates that strong, high-dimensional augmentations—particularly adapted versions of CutOut and CutMix—enable state-of-the-art performance, establishing semantic segmentation as a rigorous benchmark for semi-supervised learning regularizers.

ABSTRACT

Consistency regularization describes a class of approaches that have yielded ground breaking results in semi-supervised classification problems. Prior work has established the cluster assumption - under which the data distribution consists of uniform class clusters of samples separated by low density regions - as important to its success. We analyze the problem of semantic segmentation and find that its' distribution does not exhibit low density regions separating classes and offer this as an explanation for why semi-supervised segmentation is a challenging problem, with only a few reports of success. We then identify choice of augmentation as key to obtaining reliable performance without such low-density regions. We find that adapted variants of the recently proposed CutOut and CutMix augmentation techniques yield state-of-the-art semi-supervised semantic segmentation results in standard datasets. Furthermore, given its challenging nature we propose that semantic segmentation acts as an effective acid test for evaluating semi-supervised regularizers. Implementation at: this https URL.

Motivation & Objective

• To investigate why semi-supervised semantic segmentation lags behind semi-supervised classification in performance.
• To analyze the data distribution of semantic segmentation and assess whether the cluster assumption—central to consistency regularization—holds.
• To evaluate the impact of data augmentation strategies on semi-supervised semantic segmentation performance.
• To propose that semantic segmentation serves as a stringent benchmark for evaluating semi-supervised regularizers.

Proposed method

• The authors analyze the data distribution of semantic segmentation datasets and find no significant low-density regions separating classes, contradicting the cluster assumption.
• They adapt CutOut and CutMix augmentation techniques to create strong, high-dimensional perturbations suitable for pixel-level segmentation tasks.
• The method applies these augmentations during training to enforce consistency between predictions on original and perturbed inputs.
• The approach uses standard consistency regularization loss to optimize model predictions across augmented and original samples.
• The framework is evaluated on standard semi-supervised semantic segmentation benchmarks using standard training protocols.
• The implementation is released publicly to enable reproducibility and further benchmarking.

Experimental results

Research questions

• RQ1 Does the cluster assumption—underlying success in semi-supervised classification—hold in semantic segmentation?
• RQ2 Why do existing consistency regularization methods underperform in semi-supervised semantic segmentation?
• RQ3 Can strong, high-dimensional data augmentations like CutMix and CutOut overcome the lack of low-density regions in semantic segmentation?
• RQ4 Do these augmentations lead to state-of-the-art performance in semi-supervised semantic segmentation?
• RQ5 Can semantic segmentation serve as a robust benchmark for evaluating semi-supervised learning regularizers?

Key findings

• Semantic segmentation data distributions lack low-density regions between classes, which undermines the cluster assumption and explains poor performance of standard consistency regularization.
• Adapted CutOut and CutMix augmentations significantly improve semi-supervised semantic segmentation performance, achieving state-of-the-art results on standard benchmarks.
• The success of these augmentations is attributed to their ability to create strong, high-dimensional perturbations that enhance model robustness.
• The absence of low-density regions in segmentation data makes it a more challenging and effective testbed for evaluating semi-supervised learning methods.
• The paper establishes that semantic segmentation can serve as a rigorous acid test for semi-supervised regularizers due to its high complexity and lack of favorable geometric assumptions.