[Paper Review] Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation

Yude Wang, Jie Zhang+3 more|arXiv (Cornell University)|Apr 9, 2020

Advanced Neural Network ApplicationsComputer Science38 references57 citations

TL;DR

SEAM introduces self-supervised equivariant regularization and a pixel correlation module to refine class activation maps under image-level supervision, achieving state-of-the-art weakly supervised semantic segmentation on PASCAL VOC 2012.

ABSTRACT

Image-level weakly supervised semantic segmentation is a challenging problem that has been deeply studied in recent years. Most of advanced solutions exploit class activation map (CAM). However, CAMs can hardly serve as the object mask due to the gap between full and weak supervisions. In this paper, we propose a self-supervised equivariant attention mechanism (SEAM) to discover additional supervision and narrow the gap. Our method is based on the observation that equivariance is an implicit constraint in fully supervised semantic segmentation, whose pixel-level labels take the same spatial transformation as the input images during data augmentation. However, this constraint is lost on the CAMs trained by image-level supervision. Therefore, we propose consistency regularization on predicted CAMs from various transformed images to provide self-supervision for network learning. Moreover, we propose a pixel correlation module (PCM), which exploits context appearance information and refines the prediction of current pixel by its similar neighbors, leading to further improvement on CAMs consistency. Extensive experiments on PASCAL VOC 2012 dataset demonstrate our method outperforms state-of-the-art methods using the same level of supervision. The code is released online.

Motivation & Objective

• Motivate the gap between full and weak supervision in semantic segmentation.
• Propose self-supervised equivariant regularization to enforce consistency of CAMs across transformed inputs.
• Introduce a pixel correlation module to refine CAMs with contextually informed affinities.
• Develop a siamese network architecture with equivariant cross regularization losses to train CAMs.
• Demonstrate state-of-the-art performance on PASCAL VOC 2012 using only image-level labels.

Proposed method

• Embed equivariant regularization in a shared-weight siamese network to enforce CAM consistency under affine transforms (ER loss).
• Integrate a pixel correlation module (PCM) that refines CAMs via learned pixel affinities using a self-attention-like mechanism.
• Fuse CAM refinement with equivariant supervision through equivariant cross regularization (ECR) losses between branches.
• Handle background/foreground balance with a foreground-background scoring and a background threshold during inference.
• Train with image-level labels using multi-label soft margin loss and augment with OHEM on the ECR loss; optional CRF post-processing.

Experimental results

Research questions

• RQ1 Can consistency of CAMs across affine-transformed inputs provide self-supervision for WSSS without extra annotations?
• RQ2 Does a pixel correlation module improve CAM stability and alignment with object shapes under weak supervision?
• RQ3 What is the combined effect of equivariant regularization and PCM on CAM quality and segmentation performance?
• RQ4 Can the proposed SEAM framework achieve state-of-the-art results on PASCAL VOC 2012 with only image-level labels?
• RQ5 How do different affine transformations affect equivariant regularization efficacy?

Key findings

• SEAM improves CAM quality, achieving higher mIoU than baselines on PASCAL VOC 2012 with image-level supervision.
• The full SEAM pipeline with ER, PCM, OHEM, and optional CRF reaches up to 56.83% mIoU on the VOC train/augmentation setup.
• Using the revised CAMs with AffinityNet-based pseudo labels yields 63.61% mIoU on VOC train set, enabling strong full-supervision segmentation results.
• SEAM achieves state-of-the-art performance on VOC 2012 test set under image-level supervision, with val mIoU around 64.5 and test mIoU 65.7 in the reported tables.
• CAMs from SEAM exhibit fewer under-activations and over-activations and are more consistent under multi-scale testing.
• PCM learns boundary-sensitive affinities, producing more complete object activation coverage and more faithful shapes.