[Paper Review] Semi-supervision semantic segmentation with uncertainty-guided self cross supervision

Yunyang Zhang, Zhiqiang Gong+3 more|arXiv (Cornell University)|Mar 10, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes uncertainty-guided self cross supervision (USCS), a novel semi-supervised semantic segmentation method that uses a multi-input multi-output (MIMO) model to enable efficient self cross supervision, reducing training costs by 40.5% in parameters and 49.1% in MACs. It further improves performance by using uncertainty estimates to suppress noisy pseudo-labels during training, achieving state-of-the-art mIoU on PASCAL VOC 2012 with only 1/8 of the labeled data.

ABSTRACT

As a powerful way of realizing semi-supervised segmentation, the cross supervision method learns cross consistency based on independent ensemble models using abundant unlabeled images. However, the wrong pseudo labeling information generated by cross supervision would confuse the training process and negatively affect the effectiveness of the segmentation model. Besides, the training process of ensemble models in such methods also multiplies the cost of computation resources and decreases the training efficiency. To solve these problems, we propose a novel cross supervision method, namely uncertainty-guided self cross supervision (USCS). In addition to ensemble models, we first design a multi-input multi-output (MIMO) segmentation model which can generate multiple outputs with shared model and consequently impose consistency over the outputs, saving the cost on parameters and calculations. On the other hand, we employ uncertainty as guided information to encourage the model to focus on the high confident regions of pseudo labels and mitigate the effects of wrong pseudo labeling in self cross supervision, improving the performance of the segmentation model. Extensive experiments show that our method achieves state-of-the-art performance while saving 40.5% and 49.1% cost on parameters and calculations.

Motivation & Objective

• To address the high computational cost and noise propagation in existing cross-supervision methods for semi-supervised semantic segmentation.
• To reduce the training cost of ensemble models used in cross supervision by replacing multiple independent models with a single MIMO architecture.
• To improve model robustness against noisy pseudo-labels by leveraging prediction uncertainty as a confidence-aware supervision signal.
• To achieve state-of-the-art performance with significantly reduced resource consumption in semi-supervised semantic segmentation.

Proposed method

• Proposes a multi-input multi-output (MIMO) segmentation model that generates multiple predictions from a single forward pass, enabling efficient self cross supervision without training multiple independent models.
• Implements self cross supervision by enforcing consistency between the outputs of the MIMO model, replacing traditional cross supervision between separate ensemble models.
• Introduces uncertainty-guided learning that uses uncertainty maps to weight pseudo-label supervision, reducing the influence of low-confidence, noisy predictions.
• Employs a learnable uncertainty threshold γ to dynamically suppress regions with high uncertainty in pseudo-labels during training.
• Uses grid mixing and feature fusion strategies to enhance diversity and performance in the MIMO subnetworks.
• Applies input repetition probability ρ to balance model capacity and subnetwork independence, optimizing the trade-off between diversity and performance.

Experimental results

Research questions

• RQ1 Can a multi-input multi-output (MIMO) architecture effectively replace ensemble models in cross-supervision for semi-supervised semantic segmentation while reducing computational cost?
• RQ2 How does uncertainty estimation improve the robustness of pseudo-label supervision in semi-supervised learning?
• RQ3 What is the optimal balance between subnetwork diversity and model capacity in a MIMO-based self cross supervision framework?
• RQ4 How does the uncertainty-guided learning mechanism mitigate the negative impact of noisy pseudo-labels on model convergence?
• RQ5 To what extent can the proposed method reduce training costs while maintaining or improving segmentation accuracy?

Key findings

• The proposed method achieves state-of-the-art mIoU of 74.88% on the PASCAL VOC 2012 dataset under the 1/8 labeled data protocol, surpassing prior SOTA methods.
• The method reduces model parameters by 40.5% and MACs by 49.1% compared to standard cross-supervision methods, significantly improving training efficiency.
• Uncertainty-guided learning improves performance by 1.23% mIoU over the baseline CPS method when using ResNet-50, without additional computational cost.
• The optimal uncertainty threshold γ is found to be 0.5, where performance peaks before degrading due to over-suppression of confident predictions.
• The input repetition probability ρ = 0.4 achieves the best trade-off between subnetwork diversity and model capacity, yielding peak performance.
• Grid mixing feature fusion with block size g=1 outperforms summing and larger grid sizes, achieving the highest mIoU and non-overlap ratio.