[Paper Review] Weak-shot Semantic Segmentation via Dual Similarity Transfer

Junjie Chen, Li Niu+4 more|arXiv (Cornell University)|Oct 5, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes SimFormer, a dual similarity transfer framework for weak-shot semantic segmentation that leverages pixel-level annotations from base classes and image-level labels for novel classes. By transferring proposal-pixel similarity from base to novel classes and distilling class-agnostic pixel-pixel similarity, it enables accurate mask prediction without full supervision, achieving state-of-the-art performance on COCO-Stuff-10K and ADE20K.

ABSTRACT

Semantic segmentation is an important and prevalent task, but severely suffers from the high cost of pixel-level annotations when extending to more classes in wider applications. To this end, we focus on the problem named weak-shot semantic segmentation, where the novel classes are learnt from cheaper image-level labels with the support of base classes having off-the-shelf pixel-level labels. To tackle this problem, we propose SimFormer, which performs dual similarity transfer upon MaskFormer. Specifically, MaskFormer disentangles the semantic segmentation task into two sub-tasks: proposal classification and proposal segmentation for each proposal. Proposal segmentation allows proposal-pixel similarity transfer from base classes to novel classes, which enables the mask learning of novel classes. We also learn pixel-pixel similarity from base classes and distill such class-agnostic semantic similarity to the semantic masks of novel classes, which regularizes the segmentation model with pixel-level semantic relationship across images. In addition, we propose a complementary loss to facilitate the learning of novel classes. Comprehensive experiments on the challenging COCO-Stuff-10K and ADE20K datasets demonstrate the effectiveness of our method. Codes are available at https://github.com/bcmi/SimFormer-Weak-Shot-Semantic-Segmentation.

Motivation & Objective

• Address the high annotation cost of pixel-level masks in semantic segmentation by enabling learning of novel classes from cheaper image-level labels.
• Overcome the limitations of prior methods that require background class masks or exclude novel classes in off-the-shelf datasets.
• Develop a practical and scalable framework for dataset expansion in real-world applications.
• Improve mask quality for novel classes through similarity transfer and structural regularization.

Proposed method

• Adapt MaskFormer to disentangle semantic segmentation into proposal classification and proposal segmentation sub-tasks.
• Transfer proposal-pixel similarity from base classes to novel classes, enabling mask prediction without ground-truth masks.
• Learn and distill class-agnostic pixel-pixel similarity from base classes to regularize semantic relationships across images.
• Introduce a complementary loss to enhance mask learning by encouraging correct assignment of novel class proposals.
• Use a learnable similarity network (SimNet) to estimate pixel-level similarities between reference and query images.
• Apply knowledge distillation to transfer pixel-pixel similarity from base to novel classes, improving generalization and robustness.

Experimental results

Research questions

• RQ1 Can proposal-pixel similarity transfer from base classes effectively predict masks for novel classes with only image-level supervision?
• RQ2 Can class-agnostic pixel-pixel similarity learned from base classes improve segmentation quality and generalization for novel classes?
• RQ3 How does dual similarity transfer compare to single-supervision or baseline few-shot methods in weak-shot settings?
• RQ4 What is the impact of the complementary loss on mask learning and model robustness for novel classes?
• RQ5 How well does the method generalize when expanding to more novel classes or in real-world dataset expansion scenarios?

Key findings

• SimFormer achieves state-of-the-art performance on COCO-Stuff-10K and ADE20K, outperforming existing weak-shot and few-shot methods.
• The complementary loss alone improves mIoU by 1.8% on COCO-Stuff-10K, demonstrating its effectiveness in guiding novel class mask learning.
• Dual similarity transfer (proposal-pixel and pixel-pixel) leads to a 3.2% mIoU gain over baseline methods on COCO-Stuff-10K.
• The model shows robustness to hyperparameters β and γ, with performance stable across a reasonable range of values.
• Visualizations confirm that transferred pixel-pixel similarity captures meaningful semantic relationships across images, even for small or fine-grained objects.
• Failure cases reveal limitations in proposal classification for fine-grained classes and weak supervision for small objects, indicating room for improvement in attention and localization.