[Paper Review] SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation

Xiaolin Zhang, Yunchao Wei+2 more|arXiv (Cornell University)|Oct 22, 2018

Domain Adaptation and Few-Shot LearningComputer Science54 references84 citations

TL;DR

SG-One introduces a unified network that uses masked average pooling to create object-centered guidance vectors and cosine similarity maps to guide one-shot semantic segmentation of unseen classes, achieving state-of-the-art mean IoU on PASCAL-5i.

ABSTRACT

One-shot image semantic segmentation poses a challenging task of recognizing the object regions from unseen categories with only one annotated example as supervision. In this paper, we propose a simple yet effective Similarity Guidance network to tackle the One-shot (SG-One) segmentation problem. We aim at predicting the segmentation mask of a query image with the reference to one densely labeled support image of the same category. To obtain the robust representative feature of the support image, we firstly adopt a masked average pooling strategy for producing the guidance features by only taking the pixels belonging to the support image into account. We then leverage the cosine similarity to build the relationship between the guidance features and features of pixels from the query image. In this way, the possibilities embedded in the produced similarity maps can be adapted to guide the process of segmenting objects. Furthermore, our SG-One is a unified framework which can efficiently process both support and query images within one network and be learned in an end-to-end manner. We conduct extensive experiments on Pascal VOC 2012. In particular, our SGOne achieves the mIoU score of 46.3%, surpassing the baseline methods.

Motivation & Objective

• Motivate one-shot semantic segmentation to segment unseen categories from a single annotated example.
• Develop a robust support-object representation without changing network inputs.
• Leverage pixel-wise cosine similarity to guide query-image segmentation.
• Unify support and query processing in a single end-to-end trainable network.
• Demonstrate improved performance over prior methods on PASCAL-5i.

Proposed method

• Extract high-level features from support and query images using a shared stem network.
• Compute a robust object representation via masked average pooling over the support mask.
• Compute pixel-wise cosine similarities between the support representation and query features to form a similarity guidance map.
• Multiply query features by the similarity guidance map to steer segmentation toward the target object.
• Use a segmentation branch that ingests guidance and query features to predict the final mask in an end-to-end framework.
• Train with cross-entropy loss; enable one-shot testing without fine-tuning.

Experimental results

Research questions

• RQ1 Can a unified network with similarity-guided attention improve one-shot segmentation of unseen classes?
• RQ2 Does masked average pooling yield superior object representations over masking or concatenation approaches?
• RQ3 How does cosine-similarity guidance affect segmentation performance across PASCAL-5i folds?
• RQ4 Is the approach robust to multi-class query images and few-shot extensions (K-shot) without retraining?

Key findings

• SG-One achieves a mean IoU of 46.3% on PASCAL-5i in one-shot segmentation, outperforming baselines.
• Masked average pooling of the support mask provides a better representative vector than masking or concatenation methods.
• Five-shot results (averaging support vectors) yield 47.1% mean IoU, slightly higher than one-shot but not by a large margin.
• Compared to OSLSM and co-FCN, SG-One shows notable improvements across all four PASCAL-5i folds.
• SG-One demonstrates robustness in multi-class query scenarios, outperforming the baseline co-FCN in multi-class setups.