[Paper Review] Reliability Does Matter: An End-to-End Weakly Supervised Semantic Segmentation Approach

Bingfeng Zhang, Jimin Xiao+3 more|arXiv (Cornell University)|Nov 19, 2019

Advanced Neural Network Applications38 references22 citations

TL;DR

This paper proposes Reliable Region Mining (RRM), a simple yet effective one-step end-to-end weakly supervised semantic segmentation method that uses image-level labels to mine reliable, high-confidence pixel regions from class activation maps. By directly training a segmentation branch on these pruned, reliable regions with a novel dense energy loss, RRM achieves state-of-the-art performance (62.6 mIoU on VOC val, 62.9 on test), outperforming complex two-step methods and setting a new SOTA when extended to two-step training (66.3 mIoU val, 66.5 test).

ABSTRACT

Weakly supervised semantic segmentation is a challenging task as it only takes image-level information as supervision for training but produces pixel-level predictions for testing. To address such a challenging task, most recent state-of-the-art approaches propose to adopt two-step solutions, \emph{i.e. } 1) learn to generate pseudo pixel-level masks, and 2) engage FCNs to train the semantic segmentation networks with the pseudo masks. However, the two-step solutions usually employ many bells and whistles in producing high-quality pseudo masks, making this kind of methods complicated and inelegant. In this work, we harness the image-level labels to produce reliable pixel-level annotations and design a fully end-to-end network to learn to predict segmentation maps. Concretely, we firstly leverage an image classification branch to generate class activation maps for the annotated categories, which are further pruned into confident yet tiny object/background regions. Such reliable regions are then directly served as ground-truth labels for the parallel segmentation branch, where a newly designed dense energy loss function is adopted for optimization. Despite its apparent simplicity, our one-step solution achieves competitive mIoU scores (\emph{val}: 62.6, \emph{test}: 62.9) on Pascal VOC compared with those two-step state-of-the-arts. By extending our one-step method to two-step, we get a new state-of-the-art performance on the Pascal VOC (\emph{val}: 66.3, \emph{test}: 66.5).

Motivation & Objective

• To address the complexity and inelegance of existing two-step weakly supervised semantic segmentation methods that rely on numerous auxiliary components to generate pseudo masks.
• To improve one-step approaches by introducing a reliable, minimal supervision strategy that avoids the performance gap seen in prior end-to-end methods.
• To demonstrate that high-quality segmentation can be achieved with a simpler, more direct training pipeline using only image-level labels.
• To show that mining tiny, high-response regions from class activation maps leads to better supervision than full-object or dense pseudo-mask generation.
• To establish a new state-of-the-art performance in image-level weakly supervised semantic segmentation without using extra data or complex bells and whistles.

Proposed method

• The method uses a dual-branch network: one branch for image classification to generate class activation maps (CAMs), and a parallel segmentation branch for pixel-level prediction.
• From the CAMs, the method identifies high-response, compact regions (often tiny) corresponding to objects or backgrounds, which are considered reliable due to their high activation scores.
• These reliable regions are further refined using a Conditional Random Field (CRF) to prune noise and enhance localization accuracy.
• The pruned reliable regions are used as direct ground-truth supervision for the segmentation branch, replacing the need for full pseudo-masks.
• A novel dense energy loss is introduced, combining pixel-wise cross-entropy with a regularization term that leverages low-level features (RGB and spatial context) to improve feature learning.
• The entire network is trained end-to-end using a joint loss combining cross-entropy and dense energy loss, enabling direct optimization of segmentation performance from image-level labels.

Experimental results

Research questions

• RQ1 Can a one-step, end-to-end weakly supervised semantic segmentation method achieve competitive performance without relying on complex two-step pseudo-mask generation pipelines?
• RQ2 Does mining only the most reliable, high-activation regions from class activation maps lead to better segmentation performance than full-object or dense pseudo-mask generation?
• RQ3 Can a simple, minimal supervision strategy—using only a few high-confidence pixels—outperform more complex methods that use extensive auxiliary signals and bells and whistles?
• RQ4 How does the proposed dense energy loss, which incorporates low-level features, improve segmentation quality compared to standard cross-entropy loss in weakly supervised settings?
• RQ5 Can the proposed one-step method be extended to a two-step framework to further boost performance, and does it achieve new state-of-the-art results?

Key findings

• The one-step RRM method achieves 62.6 mIoU on the Pascal VOC validation set and 62.9 mIoU on the test set, outperforming existing end-to-end methods like EM-Adapt and setting a new SOTA for one-step approaches.
• When extended to a two-step framework using the same RRM network to generate pseudo masks, the method achieves 66.3 mIoU on validation and 66.5 mIoU on test, surpassing the previous SOTA (FickleNet at 64.9/65.3) and AffinityNet, even without extra data or auxiliary signals.
• The method outperforms AffinityNet, which uses three separate DNNs and ResNet-38 (a stronger backbone than ResNet-101), demonstrating the effectiveness of the RRM design with a single end-to-end network.
• Qualitative results show that RRM produces more accurate and boundary-aware segmentations than EM-Adapt, especially for small and complex objects.
• The ablation study confirms that the dense energy loss significantly improves performance by leveraging low-level features, and CRF refinement enhances reliability of the mined regions.
• The proposed method is simpler and more elegant than current two-step SOTA methods, which rely on object proposals, saliency maps, or multiple networks, while achieving better performance with fewer components.