[Paper Review] GuidedMix-Net: Learning to Improve Pseudo Masks Using Labeled Images as Reference

Peng Tu, Yawen Huang+3 more|arXiv (Cornell University)|Jun 29, 2021

Advanced Vision and Imaging47 references4 citations

TL;DR

GuidedMix-Net proposes a novel semi-supervised semantic segmentation method that improves pseudo mask quality by leveraging labeled images as reference through feature alignment and mutual information transfer. By generating mixed inputs from labeled and unlabeled pairs and refining features via knowledge distillation, it achieves state-of-the-art performance, improving mIoU by +7% over prior methods on benchmark datasets.

ABSTRACT

Semi-supervised learning is a challenging problem which aims to construct a model by learning from a limited number of labeled examples. Numerous methods have been proposed to tackle this problem, with most focusing on utilizing the predictions of unlabeled instances consistency alone to regularize networks. However, treating labeled and unlabeled data separately often leads to the discarding of mass prior knowledge learned from the labeled examples, and failure to mine the feature interaction between the labeled and unlabeled image pairs. In this paper, we propose a novel method for semi-supervised semantic segmentation named GuidedMix-Net, by leveraging labeled information to guide the learning of unlabeled instances. Specifically, we first introduce a feature alignment objective between labeled and unlabeled data to capture potentially similar image pairs and then generate mixed inputs from them. The proposed mutual information transfer (MITrans), based on the cluster assumption, is shown to be a powerful knowledge module for further progressive refining features of unlabeled data in the mixed data space. To take advantage of the labeled examples and guide unlabeled data learning, we further propose a mask generation module to generate high-quality pseudo masks for the unlabeled data. Along with supervised learning for labeled data, the prediction of unlabeled data is jointly learned with the generated pseudo masks from the mixed data. Extensive experiments on PASCAL VOC 2012, PASCAL-Context and Cityscapes demonstrate the effectiveness of our GuidedMix-Net, which achieves competitive segmentation accuracy and significantly improves the mIoU by +7$\%$ compared to previous state-of-the-art approaches.

Motivation & Objective

• Address the limitation of existing semi-supervised methods that treat labeled and unlabeled data separately, failing to exploit prior knowledge from labeled examples.
• Overcome the suboptimal performance of consistency-based methods that rely on manual perturbations and fail to capture structural and semantic relationships between data.
• Improve pseudo mask quality for unlabeled data by enabling feature-level interaction and knowledge transfer from labeled to unlabeled instances.
• Develop a unified training framework that jointly optimizes labeled and unlabeled data using mixed data space representations.
• Achieve competitive segmentation accuracy with significantly reduced annotation cost by leveraging limited labeled data and high-quality pseudo masks.

Proposed method

• Introduce a feature alignment objective to identify and align similar features between labeled and unlabeled image pairs, enabling the selection of relevant reference samples.
• Construct mixed inputs by combining features and inputs from aligned labeled and unlabeled pairs, creating a hybrid data space for improved generalization.
• Propose Mutual Information Transfer (MITrans), a knowledge distillation module based on the cluster assumption, to progressively refine unlabeled features using labeled knowledge.
• Design a mask generation module that produces high-quality pseudo masks for unlabeled data by leveraging the mixed data space and minimizing distribution shift from labeled masks.
• Train the model end-to-end with supervised loss on labeled data and consistency loss between original and mixed data predictions for unlabeled samples.
• Apply multi-scale testing and flipping augmentation during inference to further boost performance without additional labeled data.

Experimental results

Research questions

• RQ1 Can labeled images be effectively used as a reference to improve the quality of pseudo masks for unlabeled data in semi-supervised semantic segmentation?
• RQ2 How can structural and semantic information from labeled data be transferred to unlabeled data to enhance feature representation and segmentation accuracy?
• RQ3 Can mutual information transfer between labeled and unlabeled data improve model robustness and generalization beyond standard consistency regularization?
• RQ4 To what extent can a unified training objective that jointly optimizes labeled and unlabeled data outperform methods that treat them separately?
• RQ5 How does the proposed method compare to fully supervised models when using only a fraction of the labeled data?

Key findings

• GuidedMix-Net achieves a +7% mIoU improvement over previous state-of-the-art methods on PASCAL VOC 2012, PASCAL-Context, and Cityscapes under semi-supervised settings.
• On PASCAL VOC 2012 with only 1/8 of the labeled data, GuidedMix-Net achieves 75.2 mIoU, outperforming fully supervised FCN (69.9 mIoU) trained on the full dataset.
• With 1/2 of the labeled data, GuidedMix-Net attains 77.1 mIoU, approaching the performance of fully supervised DeepLabV3+ (78.6 mIoU) on the same split.
• Applying multi-scale testing and flipping augmentation boosts GuidedMix-Net to 78.2 mIoU on 1/2 labels, closely matching DeepLabV3+ (78.6 mIoU) with half the labeled data.
• On PASCAL-Context, GuidedMix-Net achieves mIoU gains of 14.2% and 10.3% for 1/8 and 1/4 labeled data, respectively, demonstrating strong generalization to complex, open-vocabulary scenes.
• Visual comparisons show that GuidedMix-Net produces more accurate and detailed pseudo masks, especially in contour and object boundary prediction, compared to consistency-based baselines like CCT.