[Paper Review] One-Shot Instance Segmentation

Claudio Michaelis, Ivan Ustyuzhaninov+2 more|arXiv (Cornell University)|Nov 28, 2018

Advanced Neural Network Applications99 references68 citations

TL;DR

This paper introduces Siamese Mask R-CNN to perform one-shot instance segmentation by matching a single reference image to test scenes, enabling detection and segmentation of novel categories without retraining. It establishes a baseline on MS-COCO and analyzes challenges of targeting detection to the reference category while segmentation remains more reliable.

ABSTRACT

We tackle the problem of one-shot instance segmentation: Given an example image of a novel, previously unknown object category, find and segment all objects of this category within a complex scene. To address this challenging new task, we propose Siamese Mask R-CNN. It extends Mask R-CNN by a Siamese backbone encoding both reference image and scene, allowing it to target detection and segmentation towards the reference category. We demonstrate empirical results on MS Coco highlighting challenges of the one-shot setting: while transferring knowledge about instance segmentation to novel object categories works very well, targeting the detection network towards the reference category appears to be more difficult. Our work provides a first strong baseline for one-shot instance segmentation and will hopefully inspire further research into more powerful and flexible scene analysis algorithms. Code is available at: https://github.com/bethgelab/siamese-mask-rcnn

Motivation & Objective

• Define the one-shot instance segmentation task and motivate its practical relevance for real-world scene understanding.
• Develop a model that can detect and segment novel object categories given only a single reference image.
• Evaluate the approach on MS-COCO across training-like and novel categories to establish a strong baseline.

Proposed method

• Integrate a Siamese backbone with Mask R-CNN to encode both the reference image and the query image in a shared feature space.
• Compute a similarity-based matching by averaging reference features, taking absolute differences with scene features, concatenating this to scene features, and applying a 1x1 convolution to produce task-relevant features.
• Replace the 80-way classification in Mask R-CNN with a binary match/non-match decision and use a single class-agnostic bounding box and mask head.
• Train the model episodically on MS-COCO, pre-train the backbone on a disjoint ImageNet subset, and evaluate with stochastic reference selection across four train/test splits.

Experimental results

Research questions

• RQ1 Can a metric-learning based approach enable one-shot detection and segmentation of novel categories in complex scenes?
• RQ2 How well does a Siamese extension of Mask R-CNN transfer to unseen categories compared to training categories?
• RQ3 What are the main bottlenecks when directing detection toward the reference category in one-shot instance segmentation?
• RQ4 How does the number of references (shots) affect performance on novel categories?
• RQ5 What is the impact of scene clutter on one-shot instance segmentation performance?

Key findings

• One-shot object detection with one reference achieves 37.6% mAP50 and 34.9% mAP50 for instance segmentation on training categories; using five references improves to 41.3% and 38.4%, respectively.
• For novel categories not seen during training, one-shot mAP50 is 16.3% for detection and 14.5% for segmentation; five-shot improves to 18.5% and 16.7%, respectively.
• The model shows strong bounding box and mask quality but struggles to consistently target the correct reference category, indicating overfitting to training categories.
• Performance drops in cluttered scenes with more instances, highlighting the need for robustness in crowded environments.
• The approach provides a solid baseline for one-shot instance segmentation and demonstrates generalization to unseen categories without retraining.