[Paper Review] ISIM: Iterative Self-Improved Model for Weakly Supervised Segmentation

Cenk Bircanoğlu, Nafiz Arıca|arXiv (Cornell University)|Nov 22, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes ISIM, an iterative self-improved encoder-decoder model for weakly supervised semantic segmentation that enhances Class Activation Maps (CAMs) using pseudo-segmentation labels generated via dense CRF. By jointly optimizing classification and segmentation with a modified cross-entropy loss, ISIM achieves state-of-the-art performance on Pascal VOC 2012, improving mIoU by 2.5% over prior methods using only image-level annotations.

ABSTRACT

Weakly Supervised Semantic Segmentation (WSSS) is a challenging task aiming to learn the segmentation labels from class-level labels. In the literature, exploiting the information obtained from Class Activation Maps (CAMs) is widely used for WSSS studies. However, as CAMs are obtained from a classification network, they are interested in the most discriminative parts of the objects, producing non-complete prior information for segmentation tasks. In this study, to obtain more coherent CAMs with segmentation labels, we propose a framework that employs an iterative approach in a modified encoder-decoder-based segmentation model, which simultaneously supports classification and segmentation tasks. As no ground-truth segmentation labels are given, the same model also generates the pseudo-segmentation labels with the help of dense Conditional Random Fields (dCRF). As a result, the proposed framework becomes an iterative self-improved model. The experiments performed with DeepLabv3 and UNet models show a significant gain on the Pascal VOC12 dataset, and the DeepLabv3 application increases the current state-of-the-art metric by %2.5. The implementation associated with the experiments can be found: https://github.com/cenkbircanoglu/isim.

Motivation & Objective

• To address the supervision gap in weakly supervised semantic segmentation (WSSS), where class-level labels lack spatial detail.
• To improve the coherence of Class Activation Maps (CAMs) with ground-truth segmentation by leveraging implicit image content.
• To develop an iterative, self-improved framework that alternately refines pseudo-segmentation labels and enhances CAMs without requiring pixel-level annotations.
• To achieve state-of-the-art performance on Pascal VOC 2012 using only image-level supervision and a standard encoder-decoder architecture.

Proposed method

• The framework uses a modified encoder-decoder segmentation model that jointly optimizes for classification and segmentation using only image-level labels.
• It generates pseudo-segmentation masks from CAMs using dense Conditional Random Fields (dCRF), which refine the localization and completeness of object regions.
• A pixel-wise loss is applied during training to propagate CAMs more accurately by aligning them with the pseudo-segmentation labels.
• The model iteratively improves both CAMs and segmentation predictions through end-to-end training with backpropagation.
• The architecture supports multi-task learning by sharing features between the classification head and segmentation head within a single network.
• The method is generalizable and can be applied to various backbone architectures such as ResNet, ResNeSt, and DeepLabv2/v3.

Experimental results

Research questions

• RQ1 Can iterative refinement of pseudo-segmentation labels improve the quality of Class Activation Maps (CAMs) in weakly supervised semantic segmentation?
• RQ2 How effectively can dense CRF-based post-processing enhance CAM-based segmentation without ground-truth masks?
• RQ3 To what extent can a self-improved, multi-task learning framework close the performance gap between fully supervised and weakly supervised semantic segmentation?
• RQ4 Does the proposed iterative training scheme outperform existing SOTA methods on Pascal VOC 2012 using only image-level annotations?

Key findings

• The ISIM framework achieves a mean Intersection over Union (mIoU) of 72.94% on the Pascal VOC 2012 test split using DeepLabv3 with a ResNeSt-200 backbone.
• With ResNet-101 as the backbone, ISIM achieves 70.38% mIoU on the test set, outperforming previous SOTA methods that use additional saliency map data.
• The method improves the state-of-the-art mIoU by 2.5% over prior approaches when using the same backbone (ResNet-101) and only image-level supervision.
• The ablation study confirms that the iterative refinement with dCRF and pixel-wise loss significantly improves CAM quality and segmentation accuracy.
• The framework achieves strong performance even without additional data or saliency maps, outperforming methods like EPS and SPML in the same setting.
• The proposed method is general and effective across different backbone architectures, including ResNet-50, ResNet-101, and ResNeSt-200.