[Paper Review] Prototype Refinement Network for Few-Shot Segmentation

Jinlu Liu, Yongqiang Qin|arXiv (Cornell University)|Feb 10, 2020

Domain Adaptation and Few-Shot Learning26 references23 citations

TL;DR

This paper proposes Prototype Refinement Network (PRNet), a novel few-shot segmentation method that enhances prototype representativeness through bidirectional prototype learning, model adaptation on support sets, and a parameter-free prototype fusion mechanism. PRNet achieves state-of-the-art performance, outperforming prior methods by 13.1% in mean IoU on COCO-20i in 1-shot settings.

ABSTRACT

Few-shot segmentation targets to segment new classes with few annotated images provided. It is more challenging than traditional semantic segmentation tasks that segment known classes with abundant annotated images. In this paper, we propose a Prototype Refinement Network (PRNet) to attack the challenge of few-shot segmentation. It firstly learns to bidirectionally extract prototypes from both support and query images of the known classes. Furthermore, to extract representative prototypes of the new classes, we use adaptation and fusion for prototype refinement. The step of adaptation makes the model to learn new concepts which is directly implemented by retraining. Prototype fusion is firstly proposed which fuses support prototypes with query prototypes, incorporating the knowledge from both sides. It is effective in prototype refinement without importing extra learnable parameters. In this way, the prototypes become more discriminative in low-data regimes. Experiments on PASAL-$5^i$ and COCO-$20^i$ demonstrate the superiority of our method. Especially on COCO-$20^i$, PRNet significantly outperforms existing methods by a large margin of 13.1\% in 1-shot setting.

Motivation & Objective

• Address the challenge of few-shot semantic segmentation, where models must generalize to new classes with only a few annotated support images.
• Overcome the limitation of biased and non-representative prototypes in low-data regimes, especially due to data scarcity and intra-class variance.
• Improve prototype quality for segmentation by leveraging knowledge from both support and query images through bidirectional learning and refinement.
• Develop an efficient, parameter-free refinement strategy that enhances prototype discriminability without increasing model complexity.

Proposed method

• Bidirectional prototype learning: Extracts prototypes from both support and query images during training using masked average pooling on deep features.
• Model adaptation at test time: Retrains the feature extractor on the few-shot support images to adapt to unseen classes, improving prototype relevance.
• Two-stage prototype fusion: Fuses support prototypes with query-derived prototypes using similarity maps and a self-adaptive threshold α to select confident regions.
• Iterative refinement: Applies fusion twice—first fusing support and initial query prototypes, then fusing with refined query prototypes—to progressively improve prototype quality.
• Nearest prototype matching: Segments query images by assigning each pixel to the nearest prototype in feature space, using the refined prototypes as final representations.
• Self-adaptive thresholding: Dynamically selects high-confidence regions from similarity maps to guide prototype extraction and fusion, enhancing robustness.

Experimental results

Research questions

• RQ1 Can bidirectional prototype learning from both support and query images improve prototype representativeness in few-shot segmentation?
• RQ2 How effective is model adaptation on few-shot support sets in refining prototypes for unseen classes?
• RQ3 Can a parameter-free fusion mechanism significantly improve prototype quality and segmentation accuracy?
• RQ4 Does the proposed refinement strategy generalize across different few-shot settings (e.g., 1-shot vs. 5-shot) and datasets (e.g., PASCAL-5i vs. COCO-20i)?
• RQ5 Can the method maintain strong performance in higher-way few-shot segmentation (e.g., 2-way) compared to existing one-way methods?

Key findings

• PRNet achieves a 13.1% absolute improvement in mean IoU over prior methods on COCO-20i in the 1-shot setting, setting a new state-of-the-art.
• On COCO-20i, PRNet achieves a mean IoU of 33.37% with ResNet-101, a 17.4% improvement over previous methods in the 5-shot setting.
• The proposed prototype fusion mechanism improves performance without adding any learnable parameters, demonstrating efficiency and effectiveness.
• Ablation studies show that fusion with balanced weights (ωs=0.5, ωq=0.5) yields optimal trade-off between mean-IoU and binary-IoU scores.
• In 2-way few-shot segmentation, PRNet achieves a mean IoU of 54.59%, outperforming methods that report only on 1-way tasks, indicating strong generalization to higher-way settings.
• The method consistently outperforms baselines across all folds and backbone networks (VGG, ResNet-50, ResNet-101) on both PASCAL-5i and COCO-20i datasets.