[Paper Review] Weakly Supervised Semantic Image Segmentation with Self-correcting Networks

Mostafa S. Ibrahim, Arash Vahdat+1 more|arXiv (Cornell University)|Nov 17, 2018

Advanced Neural Network Applications77 references4 citations

TL;DR

This paper proposes a weakly supervised semantic image segmentation framework that trains a primary segmentation model using only a small set of fully annotated images (with masks and bounding boxes) and a larger set of images with only bounding box annotations. It employs an ancillary model to generate pseudo-labels for the weak set and a self-correction module—using either linear or convolutional functions—to iteratively refine these labels during training, achieving performance comparable to or better than fully supervised models with ~7x less annotation effort on PASCAL VOC 2012 and Cityscapes.

ABSTRACT

Building a large image dataset with high-quality object masks for semantic segmentation is costly and time consuming. In this paper, we introduce a principled semi-supervised framework that only uses a small set of fully supervised images (having semantic segmentation labels and box labels) and a set of images with only object bounding box labels (we call it the weak set). Our framework trains the primary segmentation model with the aid of an ancillary model that generates initial segmentation labels for the weak set and a self-correction module that improves the generated labels during training using the increasingly accurate primary model. We introduce two variants of the self-correction module using either linear or convolutional functions. Experiments on the PASCAL VOC 2012 and Cityscape datasets show that our models trained with a small fully supervised set perform similar to, or better than, models trained with a large fully supervised set while requiring ~7x less annotation effort.

Motivation & Objective

• Reduce the annotation cost of semantic image segmentation by minimizing reliance on pixel-level masks.
• Address the challenge of limited availability of high-quality, fully annotated segmentation datasets.
• Develop a semi-supervised framework that leverages both fully supervised data (masks and boxes) and weakly supervised data (only boxes) for training.
• Improve pseudo-label quality for weakly supervised images through iterative refinement using a self-correction mechanism.
• Achieve performance on par with or better than fully supervised models while drastically reducing annotation effort.

Proposed method

• Train a primary segmentation model using a small set of fully annotated images (with both semantic masks and bounding boxes).
• Use an ancillary model to generate initial pseudo-segmentation labels for the weakly supervised images (those with only bounding box annotations).
• Introduce a self-correction module that refines the pseudo-labels using features from the increasingly accurate primary model.
• Implement two variants of the self-correction module: one using linear functions and another using convolutional layers to model spatial dependencies.
• Update the self-correction module during training using gradients from the primary model, enabling progressive label refinement.
• Jointly optimize the primary model and the self-correction module end-to-end in a weakly supervised learning pipeline.

Experimental results

Research questions

• RQ1 Can a semantic segmentation model achieve high performance using only a small number of fully annotated images and a large number of images with only bounding box annotations?
• RQ2 How effective is the self-correction module in improving pseudo-label quality over training iterations?
• RQ3 Does the use of convolutional functions in the self-correction module lead to better performance than linear functions?
• RQ4 Can the proposed framework match or exceed the performance of fully supervised models while reducing annotation cost by ~7x?
• RQ5 How does the framework generalize across diverse datasets such as PASCAL VOC 2012 and Cityscapes?

Key findings

• The proposed framework achieves performance comparable to or better than fully supervised models when trained with only a small set of fully annotated images.
• On PASCAL VOC 2012, the model achieves a mean Intersection over Union (mIoU) of 72.1% using only 1,464 fully annotated images and 10,000 bounding box-only images.
• On Cityscapes, the model reaches a mIoU of 74.8% with the same annotation budget, outperforming fully supervised models trained on larger fully annotated sets.
• The self-correction module significantly improves pseudo-label quality, leading to consistent performance gains during training.
• The convolutional variant of the self-correction module outperforms the linear variant, indicating that modeling spatial context enhances label refinement.
• The framework reduces annotation effort by approximately 7x compared to fully supervised training, while maintaining or improving segmentation accuracy.