[Paper Review] Prior Guided Feature Enrichment Network for Few-Shot Segmentation

Zhuotao Tian, Hengshuang Zhao+4 more|arXiv (Cornell University)|Aug 4, 2020

Domain Adaptation and Few-Shot Learning58 references48 citations

TL;DR

PFENet introduces a training-free prior mask from high-level features and a Feature Enrichment Module to adaptively fuse support information with query features, achieving state-of-the-art few-shot segmentation on PASCAL-5i and COCO without extra training cost.

ABSTRACT

State-of-the-art semantic segmentation methods require sufficient labeled data to achieve good results and hardly work on unseen classes without fine-tuning. Few-shot segmentation is thus proposed to tackle this problem by learning a model that quickly adapts to new classes with a few labeled support samples. Theses frameworks still face the challenge of generalization ability reduction on unseen classes due to inappropriate use of high-level semantic information of training classes and spatial inconsistency between query and support targets. To alleviate these issues, we propose the Prior Guided Feature Enrichment Network (PFENet). It consists of novel designs of (1) a training-free prior mask generation method that not only retains generalization power but also improves model performance and (2) Feature Enrichment Module (FEM) that overcomes spatial inconsistency by adaptively enriching query features with support features and prior masks. Extensive experiments on PASCAL-5$^i$ and COCO prove that the proposed prior generation method and FEM both improve the baseline method significantly. Our PFENet also outperforms state-of-the-art methods by a large margin without efficiency loss. It is surprising that our model even generalizes to cases without labeled support samples. Our code is available at https://github.com/Jia-Research-Lab/PFENet/.

Motivation & Objective

• Address generalization loss from over-reliance on high-level features trained on base classes in few-shot segmentation.
• Mitigate spatial inconsistency between query and support targets via adaptive feature enrichment.
• Develop a training-free prior mask generator using high-level features to guide segmentation without harming generalization.
• Propose a multi-scale Feature Enrichment Module (FEM) to refine query features with support features and priors across scales.
• Demonstrate state-of-the-art performance on standard benchmarks with efficient models and analyze components through ablations.

Proposed method

• Propose a training-free prior generation that converts ImageNet pre-trained high-level features into a pixel-wise prior mask YQ by computing the maximum cosine similarity between query and support features and normalizing the result.
• Develop a Feature Enrichment Module (FEM) that: (i) performs inter-source enrichment by merging query, support, and prior at multiple scales; (ii) enables inter-scale interaction via a top-down information path to propagate fine-scale information to coarser scales; (iii) concentrates information across scales into the refined query feature, with intermediate supervision to guide learning.
• Integrate the prior generation and FEM into the PFENet architecture, sharing a backbone (ImageNet pre-trained) to extract middle-level features for support and query, and high-level features to generate priors.
• Use 1x1 convolutions to reduce channels, multi-scale pooling to create FEM inputs, and a final classification head to produce pixel-wise predictions.
• Train with cross-entropy loss, combining intermediate losses from FEM scales and the final loss with a balancing coefficient, without updating backbone parameters.

Experimental results

Research questions

• RQ1 Can a training-free high-level feature prior improve generalization to unseen classes in few-shot segmentation?
• RQ2 Does a multi-scale Feature Enrichment Module effectively resolve spatial inconsistency between query and support samples?
• RQ3 How do high-level priors and FEM interact to improve performance on standard few-shot segmentation benchmarks?
• RQ4 Is PFENet able to outperform existing methods on PASCAL-5i and COCO while maintaining efficiency and generalization?
• RQ5 Can the approach generalize to settings with limited or no labeled support samples?

Key findings

• PFENet achieves state-of-the-art results on PASCAL-5i and COCO without efficiency loss.
• The training-free prior generation preserves generalization and boosts segmentation accuracy by leveraging high-level features in a class-insensitive manner.
• FEM provides superior gains over traditional multi-scale structures (PPM, ASPP) by enabling conditioned inter-scale and inter-source interactions.
• PFENet shows strong performance with few learnable parameters (as low as 10.4M for VGG-based models, 10.8M for ResNet-based models).
• The model demonstrates non-trivial performance in zero-shot-like scenarios where no labeled support samples are available, highlighting robustness of the prior and enrichment approach.